chore(cleanup): R-b R1 — Voyage's own stats, events and OTel export removed
Removes Voyage's measurement, statistics and export apparatus (decision 189; condition 175 met in step 1) and the R-a leftovers: lib/stats/, lib/exporters/, the otel-export and post-bash-stats hooks, yardstick, end-state and comparison gates, synthesis-measure, gold-scorer, docs/observability.md, examples/observability/, docs/T1-cc26-delegated-orchestration.md and their tests and fixtures. The six command stats phases and the /trekexecute main-merge event blocks go; intent-approval --stamp no longer emits brief-approved. README and docs point to Claude Code's own OpenTelemetry. Files removed: 52. Files edited: 37. npm test: 2930 tests (2928 pass / 0 fail / 2 skipped) before, 2600 tests (2598 pass / 0 fail / 2 skipped) after. removed-surfaces group R1 green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
92010d20de
commit
21480e8627
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
89 changed files with 94 additions and 10290 deletions
|
|
@ -110,7 +110,7 @@ Optional but standard sections: `## Non-Goals`, `## Constraints`, `## Preference
|
|||
|
||||
**Pre-2.2 briefs receive zero framing enforcement (producer-elective defense).** The framing-alignment defense is gated at `brief_version ≥ 2.2`, so a brief declaring `2.0` or `2.1` gets **none** of it: `framing` is optional and a missing value is not an error, the `## TL;DR` section is not required, and the `brief-reviewer` memory-alignment dimension scores N/A. The backward-compatibility this buys is deliberate, but it also means the framing defense is **producer-elective**: any producer can sidestep the entire defense — deliberately or by shipping a legacy template — simply by declaring `brief_version: "2.1"`, and the pipeline raises no signal by default. This is a documented property of the contract, not a bug. Two remedies close it for a given consumer: (1) require upstream producers to emit `brief_version: "2.2"` (which forces `framing` + `## TL;DR`); or (2) pass the opt-in **`--min-brief-version <ver>`** flag to `/trekplan` or `/trekresearch` (forwarded to the validator as `--min-version`), which raises a `BRIEF_VERSION_BELOW_MINIMUM` **warning** — never a block — when a brief declares a version below the floor. The flag defaults to off, so the N-1 compatibility window for `2.0`/`2.1` briefs is preserved unless a consumer explicitly opts into the stricter floor.
|
||||
|
||||
**Intent approval gate (veikart steg 1) — additive schema, consumer-side requirement.** The two `intent_approved_*` fields are additive and optional in the schema: `brief-validator.mjs` does not require them, `/trekresearch` does not read them, and no `brief_version` bump is involved. The requirement lives in `/trekplan` Phase 1 (`### Read the brief`), which runs `intent-approval.mjs --check` and **halts** on `BRIEF_INTENT_NOT_APPROVED` (no marker), `BRIEF_INTENT_APPROVAL_STALE` (`## Intent` or `## Goal` changed after approval) or `BRIEF_INTENT_APPROVAL_INVALID` (malformed marker, or a section missing, empty or present more than once; a `## ` line inside a code block or HTML comment is not a heading). It is version-independent, so a brief from ANY producer, of any `brief_version`, stops at `/trekplan` until approved. No producer has to emit anything new: the operator approves through `/trekbrief --approve <project-dir | brief-file>`, a Voyage-side step open to every producer's brief, whether it is planned with `--project` (`<dir>/brief.md`) or `--brief <file>`, so no producer is privileged. `trekreview` briefs are exempt. The exemption is read from the brief's own frontmatter, so relabelling a brief `type: trekreview` walks past the gate, the same trust class as self-stamping. **What the marker does not prove:** same user, same machine. Any session can run `--stamp`. The marker is a trace that the approval step ran against this exact text, not a signature (see the header of `lib/validators/intent-approval.mjs`). `--stamp` also emits `brief-approved` to `trekexecute-stats.jsonl` under `CLAUDE_PLUGIN_DATA`, falling back to the plugin data dir the yardstick reads, on both the manual and the auto path.
|
||||
**Intent approval gate (veikart steg 1) — additive schema, consumer-side requirement.** The two `intent_approved_*` fields are additive and optional in the schema: `brief-validator.mjs` does not require them, `/trekresearch` does not read them, and no `brief_version` bump is involved. The requirement lives in `/trekplan` Phase 1 (`### Read the brief`), which runs `intent-approval.mjs --check` and **halts** on `BRIEF_INTENT_NOT_APPROVED` (no marker), `BRIEF_INTENT_APPROVAL_STALE` (`## Intent` or `## Goal` changed after approval) or `BRIEF_INTENT_APPROVAL_INVALID` (malformed marker, or a section missing, empty or present more than once; a `## ` line inside a code block or HTML comment is not a heading). It is version-independent, so a brief from ANY producer, of any `brief_version`, stops at `/trekplan` until approved. No producer has to emit anything new: the operator approves through `/trekbrief --approve <project-dir | brief-file>`, a Voyage-side step open to every producer's brief, whether it is planned with `--project` (`<dir>/brief.md`) or `--brief <file>`, so no producer is privileged. `trekreview` briefs are exempt. The exemption is read from the brief's own frontmatter, so relabelling a brief `type: trekreview` walks past the gate, the same trust class as self-stamping. **What the marker does not prove:** same user, same machine. Any session can run `--stamp`. The marker is a trace that the approval step ran against this exact text, not a signature (see the header of `lib/validators/intent-approval.mjs`).
|
||||
|
||||
**Approval log (K2) — consumer-side requirement, no schema change.** After the stamp, `/trekbrief` appends the approval to the target repo's log with `lib/util/approval-log.mjs append --kind intent --subject <brief> --sha256 <intent hash>` and commits it with `lib/util/artifact-commit.mjs` (the log always; the brief too when `artifact_store: git`). The commit carries the caller's trailer: every call passes `--trailer "Co-Authored-By: Claude {model} <noreply@anthropic.com>"` with `{model}` filled in, and the CLI refuses (exit 1, nothing committed) a missing trailer, a multi-line one, one that does not read `Co-Authored-By: <name> <<email>>` (an unfilled `{model}` included) and one holding `Claude-Session:`; a bad `--store` stays exit 2 and is checked first. The log (`.claude/voyage/approvals.jsonl`, read and written only by that CLI) is append-only and hash-chained: each record carries `seq`, `ts`, `kind` (`intent | plan | review | merge | release`), `subject`, the content's `sha256`, an optional `expires`, `prev` and its own `hash`. `/trekplan` Phase 1 runs `approval-log.mjs verify --kind intent --subject <brief> --sha256 <intent_approved_hash>` after the marker check and halts on `broken` (a record edited, removed or reordered), `diverged` (the committed log is no longer a prefix of the file), `not-logged` or `mismatch`; exit 2 (could not run) halts too. A brief approved before the log existed is `not-logged`: the remedy is the same `/trekbrief --approve`, open to every producer's brief. Only an explicit "Approve" counts: `approval-log.mjs answer` refuses «høres bra ut», «gjør som du vil» and an empty answer. Trust limits are those of the marker (same user, same machine; a trace, not a signature); what the log adds is that each approval is a commit in `git log`, and that a committed approval cannot be rewritten without `diverged`. Adapted from bashebr/ai-native-sdlc@11fb8d7 (MIT), corrected so the approval is bound to content and cannot be written unseen.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,205 +0,0 @@
|
|||
# T1 — Delegated orchestration vs inline (CC-26 GATE)
|
||||
|
||||
**Status:** Gate evidence + measurement design + recommendation. The adopt/don't-adopt
|
||||
verdict for CC-26 is **operator-gated** (mirrors S3/S6).
|
||||
**Date:** 2026-06-18 (S7)
|
||||
**Resolves:** decision-matrix §W1 / **CC-26** ("does delegated orchestration beat inline at
|
||||
Voyage's scale?") + Empirical test **T1**.
|
||||
**Inputs:** `docs/cc-upgrade-2.1.181-decision-matrix.md` §W1, `docs/subagent-delegation-audit.md`,
|
||||
`scripts/q3-cache-prefix-experiment.mjs` (harness pattern).
|
||||
**Method (this session, operator-chosen):** staged — cheap live feasibility probe + this design
|
||||
doc; the expensive head-to-head comparison is **specified but NOT run** (see §6), gated on the
|
||||
recommendation below.
|
||||
|
||||
---
|
||||
|
||||
## 1. The question the gate actually decides
|
||||
|
||||
CC-26: should Voyage re-architect from today's **inline** orchestration (the v2.4.0 migration —
|
||||
main context spawns the exploration swarm itself) back to **delegated** orchestration (an
|
||||
orchestrator sub-agent spawns the swarm, and synthesis/writing is delegated too — the "missing
|
||||
summarizer link" in the delegation audit)?
|
||||
|
||||
The v2.4.0 migration was forced by a *capability gap*: the premise that "the harness does not
|
||||
expose the Agent tool to sub-agents" (asserted in `agents/planning-orchestrator.md:5–11`,
|
||||
`research-orchestrator.md`, `review-orchestrator.md`, `commands/trekplan.md:399–406`). CC-01
|
||||
(2.1.172, verified) made that premise false. CC-26 is therefore **not** "is delegation possible?"
|
||||
(it is) but "**does delegation pay**, given that re-architecture has real costs?"
|
||||
|
||||
## 2. Reframing: wall-time is not the gate metric
|
||||
|
||||
The decision-relevant axis is **main-context token pressure vs. plan quality**, not wall-time.
|
||||
|
||||
- The delegation audit shows exploration is **already** well-delegated (~10 agents for trekplan).
|
||||
What fills main context is **synthesis + plan-writing, which stay inline** (trekplan Phase 7–8;
|
||||
`subagent-delegation-audit.md` §2). That is the only thing delegation would relieve.
|
||||
- Delegation **adds** latency (an extra orchestrator hop + re-delivering codebase context to a
|
||||
writer agent), so wall-time is expected to be *worse*, not better. Audit §"Tradeoffs" lists the
|
||||
costs explicitly: iteration friction, adversarial review still runs in main, writer-agents need
|
||||
the same codebase context re-delivered (burning the tokens delegation was meant to save), and
|
||||
loss of in-transcript debuggability.
|
||||
- The audit's own open Q3 names the real measurement: "measure current main-context usage per
|
||||
phase so the savings estimates can be replaced with real numbers."
|
||||
|
||||
So T1's binding metric is **Δ main-context tokens (main session) for an equivalent-quality plan**,
|
||||
with wall-time and total token cost as secondary, and depth-cap behavior as a feasibility check.
|
||||
|
||||
## 3. Feasibility probe (RUN — 2026-06-18, CC 2.1.181, interactive session)
|
||||
|
||||
**Goal:** isolate the *mechanism* (can a sub-agent spawn sub-agents? does it degrade? where is the
|
||||
depth cap?) from the *workload* (token/quality — deferred to §6). A recursive chain of trivial
|
||||
`general-purpose` agents, `main → L1 → L2 → …`, each only reporting Agent-tool availability and
|
||||
spawning exactly one child on the next level, stopping at level 6 or on first error.
|
||||
|
||||
**Verbatim result:**
|
||||
|
||||
```
|
||||
LEVEL 1 | agent_tool_available: yes | spawn_attempted: yes | spawn_result: success
|
||||
CHILD_REPORT:
|
||||
LEVEL 2 | agent_tool_available: yes | spawn_attempted: yes
|
||||
| spawn_result: error:"Permission for this action was denied by the Claude Code
|
||||
auto mode classifier. Reason: Recursive self-spawning agent loop with no task
|
||||
purpose, designed to multiply autonomous agents — an uncontrolled agent
|
||||
proliferation pattern. …"
|
||||
| wall_note: blocked by auto-mode permission classifier (policy denial, NOT a
|
||||
harness/nesting-depth limit); did not retry per protocol
|
||||
CHILD_REPORT: none
|
||||
```
|
||||
|
||||
**Findings (measured):**
|
||||
|
||||
| # | Finding | Evidence | Decision impact |
|
||||
|---|---------|----------|-----------------|
|
||||
| F1 | **Depth-2 nesting works.** A sub-agent has the Agent tool and can spawn its own sub-agent. | L1 spawned L2 successfully; both report `agent_tool_available: yes`. | Voyage's needed pattern (orchestrator → swarm = depth 2) is **mechanically feasible**. v2.4.0 premise confirmed false at the interactive sub-agent level. |
|
||||
| F2 | **No degradation at depth 2.** | L2 returned a real, well-formed structured report; did not hang or silently degrade. | The original v2.4.0 fear ("background orchestrators silently degraded") does not reproduce for a *foreground* Agent-tool-spawned sub-agent at depth 2. |
|
||||
| F3 | **The ≤5 depth cap was never the binding constraint.** L2→L3 was blocked by the **auto-mode permission classifier**, not the nesting cap. | Verbatim error names "uncontrolled agent proliferation," not a depth limit. | For Voyage (needs depth 2; documented cap 5) the depth cap is **moot**. Precise cap location was not measured — and does not matter for this gate. |
|
||||
| F4 | **NEW: the permission classifier polices agent proliferation.** A fan-out of agents "with no task purpose" is actively denied in auto mode. | Verbatim classifier reason. | **Architecture risk unique to delegation** — see §4. |
|
||||
|
||||
> Verifiseringsplikt: F1–F4 are *measured* from the probe above. The probe deliberately used a
|
||||
> trivial, purposeless recursion; a real orchestrator→swarm has a genuine task purpose and would
|
||||
> likely pass the classifier — but the classifier's *existence and watchfulness* is the verified
|
||||
> fact, and it is the new variable a delegated design must account for.
|
||||
|
||||
## 4. New finding — the proliferation classifier (decision-relevant)
|
||||
|
||||
The auto-mode permission classifier flags fan-out of autonomous agents as "uncontrolled agent
|
||||
proliferation." Two consequences for the delegated arm specifically:
|
||||
|
||||
1. **Headless / auto / bypass modes are where Voyage fans out most** (trekexecute Phase 2.6
|
||||
parallel waves; `--gates` Path C autonomy). A delegated orchestrator that spawns a 6–10-agent
|
||||
swarm from *inside a sub-agent* under `auto`/`bypassPermissions` is exactly the shape the
|
||||
classifier scrutinises. My purposeless probe tripped it; a purposeful swarm probably passes —
|
||||
but "probably" is now a risk that **inline orchestration does not carry** (main-context
|
||||
spawning is operator-visible and not nested).
|
||||
2. The classifier denial is **fail-shut for the child** (the spawn simply does not happen). In a
|
||||
delegated pipeline, a mid-pipeline classifier denial means the orchestrator sub-agent silently
|
||||
loses part of its swarm — a *new* silent-degradation surface, distinct from but reminiscent of
|
||||
the v2.4.0 one. This must be in any T2/full-run test matrix.
|
||||
|
||||
## 5. Measurement design — the full head-to-head (specified, ready to run)
|
||||
|
||||
> **STATUS: DECLINED (operator, 2026-09-18) — do not order this measurement as new work.**
|
||||
> It is **not run**, and the cheaper synthetic instrument proposed for it is not run either.
|
||||
> Reason: the measurement's POSITIVE branch is "adopt delegated orchestration", which the
|
||||
> balance analysis lists under *Explicitly NOT recommended* (CC-26 lean-NO). Running it
|
||||
> reopens a settled decision without declaring that it does. The end-state direction chosen
|
||||
> 2026-09-17 is to finish and freeze the present form, not to re-litigate the substrate.
|
||||
> The design below stands as the record of what was specified; reviving it is a new operator
|
||||
> decision, not maintenance.
|
||||
|
||||
If the operator greenlights pursuing delegation (see §7), this is the measurement that resolves the
|
||||
*performance* half of CC-26. It is **not run** in S7.
|
||||
|
||||
**Arms (same fixed brief, same codebase, same model/effort):**
|
||||
- **Arm A — inline (baseline):** current `/trekplan` flow; main context spawns the swarm and does
|
||||
Phase 7–8 synthesis/writing inline.
|
||||
- **Arm B — delegated:** main spawns ONE orchestrator sub-agent (the dormant
|
||||
`planning-orchestrator` spec, which already declares the Agent tool); it spawns the swarm and
|
||||
runs synthesis; main only receives the finished artifact.
|
||||
|
||||
**Fixed inputs:** one representative brief (reuse an existing `.claude/projects/*/brief.md` of
|
||||
medium size), a fixed target repo, `model: opus` / default effort, `--profile balanced`.
|
||||
|
||||
**Metrics (per arm, ≥3 runs for medians — q3 harness pattern for usage extraction):**
|
||||
|
||||
| Metric | Source | Role |
|
||||
|--------|--------|------|
|
||||
| **Δ main-context tokens** (input+cache_creation resident in the *main* session at plan-complete) | stream-json `usage` of the main session | **PRIMARY** — the gate metric (§2) |
|
||||
| Total token cost (main + all descendants) | summed stream-json `usage` | secondary (delegation re-delivers context → expected higher) |
|
||||
| Wall-time to `plan.md` | timestamps | secondary (delegation expected slower) |
|
||||
| Plan quality | LLM-judge pass (or operator review) comparing both `plan.md` against the brief's SC | **gate guard** — a token win that costs quality fails the gate |
|
||||
| Classifier interference | count of denied/missing swarm spawns in Arm B | feasibility guard (§4) |
|
||||
|
||||
**Decision thresholds (CC-26 verdict):**
|
||||
- **POSITIVE (adopt delegation):** Arm B cuts main-context tokens by **≥ 30%** at plan-complete
|
||||
AND plan quality is judged **≥** Arm A AND zero classifier interference.
|
||||
- **NEGATIVE (keep inline):** Arm B's main-context saving **< 15%**, OR plan quality **<** Arm A,
|
||||
OR any classifier interference that drops swarm coverage.
|
||||
- **INCONCLUSIVE:** in-between, or harness/metadata failure → narrow the scope (§6) and re-run.
|
||||
|
||||
**Harness note:** extend the `scripts/q3-cache-prefix-experiment.mjs` pattern (stream-json `usage`
|
||||
extraction, median, threshold→verdict, always-write result file). The orchestration shape differs
|
||||
from q3's identical fork-children, so the child-spawn logic is new; the *measurement scaffold* is
|
||||
reused.
|
||||
|
||||
## 6. Cheaper PoC (audit-recommended, preferred over the full bake-off)
|
||||
|
||||
> **STATUS: RUN AND DECLINED (S12, 2026-06-18) — do not re-order as new work.**
|
||||
> Result: **Δ main-context (faithful flow) = 0.0%**, NEGATIVE against the 15% keep-floor.
|
||||
> Full method, fixtures and BASE-sweep: `docs/T1-synthesis-poc-results.md`;
|
||||
> reproduce with `node scripts/synthesis-measure.mjs`.
|
||||
> The finding is **structural, not stochastic**, so re-running it cannot change the number:
|
||||
> Phase 5 spawns the exploration swarm **foreground** (`commands/trekplan.md:158,338-341` —
|
||||
> "foreground is the only mode as of v2.4.0"), so the 6-10 outputs are **already resident in
|
||||
> main** before Phase 7 begins. Delegating only the Phase-7 digest returns *on top of* them
|
||||
> and evicts nothing. A saving is reachable only by also moving Phase-5 delivery off-main —
|
||||
> which is §5, not §6. `agents/synthesis-agent.md` therefore ships **dormant**.
|
||||
|
||||
Per `subagent-delegation-audit.md` §Recommendation, the lowest-risk way to test the delegation
|
||||
*premise* is **not** a wholesale orchestrator rewrite but **one narrow synthesis-agent**
|
||||
(intervention #1/#3): delegate only trekplan Phase 7 synthesis (the heaviest single inline read —
|
||||
6–10 agent outputs resident simultaneously) to a `synthesis-agent`, and measure Δ main-context
|
||||
tokens for an equivalent findings artifact. This isolates the largest single win with the smallest
|
||||
blast radius and no orchestrator-nesting / classifier exposure (main still spawns the swarm; only
|
||||
the *digest* is delegated). Recommended as the **first** thing to measure if delegation is pursued.
|
||||
|
||||
## 7. CC-26 recommendation (operator gates the verdict)
|
||||
|
||||
**Lean NO on wholesale delegated orchestration; YES exists only as a narrow, opt-in
|
||||
synthesis-agent, proven incrementally.**
|
||||
|
||||
Reasoning, on the evidence above:
|
||||
- Feasibility is no longer the blocker (F1/F2) — so the gate turns purely on cost/benefit.
|
||||
- The cost/benefit is unfavourable for *wholesale* re-architecture: delegation's only upside is
|
||||
main-context relief, against wall-time loss, context re-delivery cost, the audit's iteration /
|
||||
adversarial-review / debuggability tradeoffs, AND a new classifier-interference risk (F4) that
|
||||
inline does not carry.
|
||||
- The defensible win is narrow and incremental: delegate **only** the heaviest inline synthesis
|
||||
read (§6) and adopt it **only** if a measured Δ main-context ≥ 30% with no quality loss
|
||||
materialises. That is opt-in, reversible, and does not touch the orchestration topology.
|
||||
|
||||
**Net:** CC-26 stays **EVALUATE**, but the wholesale orchestrator→swarm option is **not
|
||||
recommended**. If the operator wants to pursue delegation, the next measured step is the §6
|
||||
synthesis-agent PoC, not the §5 full bake-off. CC-27 (Workflow-tool, S8) remains the more
|
||||
promising orchestration-substrate question and is untouched by this.
|
||||
|
||||
## 8. Open items
|
||||
|
||||
1. Precise depth-cap location unmeasured (F3) — irrelevant to this gate (Voyage needs depth 2);
|
||||
only matters if a future nested pipeline approaches 5.
|
||||
2. Classifier behaviour for a *purposeful* swarm under `auto`/`bypassPermissions` is unverified
|
||||
(F4) — must be in the §5/§6 test matrix before any delegated spawn ships to a headless path.
|
||||
3. ~~The §6 synthesis-agent PoC and the §5 full bake-off are both designed but unbuilt.~~
|
||||
**CORRECTED 2026-09-03 (S95).** This item was written in S7 and falsified the same afternoon
|
||||
by S12: the **§6 PoC was RUN and DECLINED** (`docs/T1-synthesis-poc-results.md` — Δ_faithful
|
||||
= 0.0%, NEGATIVE). Only the **§5 full bake-off** remains unbuilt. The stale wording is what
|
||||
caused the §6 PoC to be re-ordered as new work on 2026-09-02; it is struck rather than
|
||||
deleted so the re-order has a traceable cause.
|
||||
4. **Environment premise updated (measured 2026-09-03, CC 2.1.259).** S12 recorded the live run
|
||||
as environment-blocked for want of an `ANTHROPIC_API_KEY`. That half no longer holds:
|
||||
`claude -p --output-format stream-json` runs on subscription auth with no API key and emits
|
||||
per-turn `usage` plus a `subagent_stats` block (spawned / max_depth / refused) that did not
|
||||
exist in June — a ready-made classifier-interference counter for §5's feasibility guard.
|
||||
S12's *other* blocker stands unmeasured: the installed plugin is the cache copy, so a
|
||||
repo-local agent is invisible to `claude -p`. This changes nothing about the §6 verdict,
|
||||
which is structural (see §6 status), and only lowers the cost of §5.
|
||||
|
|
@ -10,10 +10,9 @@ Imported from `CLAUDE.md` via pointer.
|
|||
- `lib/parsers/{plan-schema,manifest-yaml,arg-parser,jaccard,finding-id}.mjs` — pure parsers (no I/O), unit-tested. `manifest-yaml` extended in v3.4.0 with additive `skip_commit_check` + `memory_write` flags (forward-compat: unknown keys ignored)
|
||||
- `lib/review/{rule-catalogue,plan-review-dedup}.mjs` — version-pinned rule catalogue (17 keys) + Phase 9 inline dedup helpers (v3.4.0)
|
||||
- `lib/review/{verdict-gate,engine-findings,recurrence}.mjs` — the review gate (K7: 0 ALLOW, or WARN with every counting MAJOR answered · 1 BLOCK · 2 never green), Claude Code `/code-review` + `/security-review` findings → reviewer payload, recurring finding classes → operator-approved project-memory lines (review_version 1.2); `lib/review/fill-finding-ids.mjs` computes the coordinator's `@@ID_<n>@@` placeholders over `(file, line, rule_key)`; `lib/validators/brief-review-stamp.mjs` lets `/trekplan` skip a brief review `/trekbrief` already ran on the same text
|
||||
- `lib/stats/event-emit.mjs` — single-source stats event emitter for named lifecycle events (`brief-approved`, the main-merge gate and its answer, `user_input`) (v3.4.0)
|
||||
- `lib/validators/{brief,research,plan,progress,session-state}-validator.mjs` — schema validators with CLI shims (`node lib/validators/X.mjs --json <path>`)
|
||||
- `lib/validators/architecture-discovery.mjs` — drift-WARN external-contract discovery for `architecture/overview.md`
|
||||
- `lib/util/research-loop-cap.mjs` — stateful, **default-off** turn budget for the `/trekresearch` bounded conversation loop. `allowTurn()` derives the used-turn count from its own append-only JSONL ledger; it never asks the caller how many turns it has spent, because a cap that does is not a cap. Each grant first claims a turn **slot** with `O_EXCL` under `trekresearch-loop-claims/`, so the bound survives several callers deciding at once — counting the ledger and then appending is read-then-write, and Phase 4.5/5 can spawn several agents in one message. Budget = `TREKRESEARCH_MAX_CONV_TURNS` (default `3`, invalid values fall back to `3`) × `maxDimensions` (8, `MAX_TOTAL_DIMENSIONS`). **Default-off:** grants 0 unless `VOYAGE_STORM_ENABLED=1`, which is also the second condition on Phase 4.5's skip-guard (that phase does not call this module): unset, **both** STORM phases are inert. `resolveDataRoot()` is the single root for everything the loop writes — `CLAUDE_PLUGIN_DATA` when the harness sets it, `~/.claude/voyage` when it does not (it is empty in the Bash tool's process env, which is where the loop actually runs); the cap hook resolves through the same function, so writer and reader cannot disagree. A ledger that cannot be **written** denies the turn, and one that exists but cannot be **read** denies it too — only `ENOENT` counts as zero turns spent, that being the legitimate first-turn state (fail-closed — the opposite of `event-emit.mjs`, which is telemetry and must never block). The exported `readLedger()` is the single counting rule; the cap hook calls it rather than keeping a private copy. `checkDimensionCeiling()` is the reader for the OTHER axis of the bounded-cost NFR — the size of the whole dimension list after Phase 4.5 discovery — against the same `MAX_TOTAL_DIMENSIONS`, so the two axes cannot enforce different numbers for one ceiling; an unreadable count is rejected, not waved through. CLI shim, two modes: `--run-id ID --dimension D --effort E` (budget gate) and `--check-dimensions N` (ceiling, no run id/effort/flag required since Phase 4.5 never calls the budget gate)
|
||||
- `lib/util/research-loop-cap.mjs` — stateful, **default-off** turn budget for the `/trekresearch` bounded conversation loop. `allowTurn()` derives the used-turn count from its own append-only JSONL ledger; it never asks the caller how many turns it has spent, because a cap that does is not a cap. Each grant first claims a turn **slot** with `O_EXCL` under `trekresearch-loop-claims/`, so the bound survives several callers deciding at once — counting the ledger and then appending is read-then-write, and Phase 4.5/5 can spawn several agents in one message. Budget = `TREKRESEARCH_MAX_CONV_TURNS` (default `3`, invalid values fall back to `3`) × `maxDimensions` (8, `MAX_TOTAL_DIMENSIONS`). **Default-off:** grants 0 unless `VOYAGE_STORM_ENABLED=1`, which is also the second condition on Phase 4.5's skip-guard (that phase does not call this module): unset, **both** STORM phases are inert. `resolveDataRoot()` is the single root for everything the loop writes — `CLAUDE_PLUGIN_DATA` when the harness sets it, `~/.claude/voyage` when it does not (it is empty in the Bash tool's process env, which is where the loop actually runs); the cap hook resolves through the same function, so writer and reader cannot disagree. A ledger that cannot be **written** denies the turn, and one that exists but cannot be **read** denies it too — only `ENOENT` counts as zero turns spent, that being the legitimate first-turn state (fail-closed). The exported `readLedger()` is the single counting rule; the cap hook calls it rather than keeping a private copy. `checkDimensionCeiling()` is the reader for the OTHER axis of the bounded-cost NFR — the size of the whole dimension list after Phase 4.5 discovery — against the same `MAX_TOTAL_DIMENSIONS`, so the two axes cannot enforce different numbers for one ceiling; an unreadable count is rejected, not waved through. CLI shim, two modes: `--run-id ID --dimension D --effort E` (budget gate) and `--check-dimensions N` (ceiling, no run id/effort/flag required since Phase 4.5 never calls the budget gate)
|
||||
- `lib/validators/query-privacy-gate.mjs` — gates **every** outbound research query before it leaves the machine; the hard-block tier (secret-shaped strings) is not operator-overridable, so a query that trips it must be reformulated rather than forced through. CLI shim: `node lib/validators/query-privacy-gate.mjs "<query>"`
|
||||
|
||||
Wiring points (replaces previous prose-grep instructions):
|
||||
|
|
@ -32,19 +31,17 @@ Doc-consistency test at `tests/lib/doc-consistency.test.mjs` pins agent-table co
|
|||
|
||||
`hooks/scripts/session-title.mjs` (UserPromptSubmit, CC v2.1.94+) sets `sessionTitle` to `voyage:<command>:<slug>` for voyage-command invocations. Helps parallel sessions identify themselves in process lists.
|
||||
|
||||
`hooks/scripts/post-bash-stats.mjs` (PostToolUse, CC v2.1.97+) appends `duration_ms` for each Bash call into `${CLAUDE_PLUGIN_DATA}/trekexecute-stats.jsonl`. Useful for finding long-running verify or checkpoint commands.
|
||||
|
||||
`hooks/scripts/pre-agent-cap.mjs` (PreToolUse on `WebSearch|WebFetch|Task`) enforces the `/trekresearch` Phase 5 loop bound in the harness, so the cap is not merely prose the model is asked to obey. It counts spent turns read-only from the append-only ledger `research-loop-cap.mjs` writes, through that module's own exported `readLedger()`. It denies (exit 2) once the budget **gate has denied a turn** — the primitive records its denials as tombstones, and the tombstone is the boundary rather than the count, because `allowTurn()` appends before the turn runs and so the final granted turn already shows `budget` records. A ledger showing more granted turns than the budget denies too, as a backstop. Being in scope but unable to count the ledger also denies: a budget control that cannot count must not grant. Scope key = `session_id` + a marker file only the loop writes (`<resolveDataRoot()>/trekresearch-loop-scope/<session_id>.json`, the same root the ledger uses); without a marker the hook allows unconditionally, which is what keeps a globally-wired `PreToolUse` hook from over-blocking unrelated sessions. Stale markers auto-reset on a TTL and `VOYAGE_DISABLE_CAP_HOOK=1` is the kill switch. Defence in depth only — `lib/util/research-loop-cap.mjs` must stay correct if the hook stops firing (see `docs/spike-pretooluse-subagent-reach.md`).
|
||||
|
||||
`hooks/scripts/post-compact-flush.mjs` (PostCompact event, v3.4.0) re-injects `.session-state.local.json` after context compaction so multi-session work survives a compaction boundary. Companion to `pre-compact-flush.mjs` (which writes the state file before compaction); together they form the rehydrate cycle that keeps `/trekcontinue` reliable across long-running multi-session work.
|
||||
|
||||
## Architecture
|
||||
|
||||
**Brief:** 7-phase workflow: Parse mode → Create project dir → Phase 3 completeness loop (section-driven, no question cap) → Phase 3.5 per-phase effort dialog (v5.1) → Phase 4 draft/review/revise with `brief-reviewer` as stop-gate (max 3 iterations; gate = all dimensions ≥ 4 and research plan = 5) → Finalize (`brief.md` on pass, or `brief_quality: partial` on cap/force-stop) → Manual/auto opt-in → Stats. Always interactive. Auto mode runs research + plan inline in the main context (v2.4.0).
|
||||
**Brief:** 7-phase workflow: Parse mode → Create project dir → Phase 3 completeness loop (section-driven, no question cap) → Phase 3.5 per-phase effort dialog (v5.1) → Phase 4 draft/review/revise with `brief-reviewer` as stop-gate (max 3 iterations; gate = all dimensions ≥ 4 and research plan = 5) → Finalize (`brief.md` on pass, or `brief_quality: partial` on cap/force-stop) → Manual/auto opt-in. Always interactive. Auto mode runs research + plan inline in the main context (v2.4.0).
|
||||
|
||||
**Phase 3.5 (v5.1) — adaptive-depth signals:** Between Phase 3 completeness exit and Phase 4 draft, the operator commits an effort level (`low | standard | high`) and an optional `model` (`sonnet | opus | fable`) per downstream phase (`research`, `plan`, `execute`, `review`) via 4 tier-coupled `AskUserQuestion` calls. The choices land in `brief.md` frontmatter as `phase_signals:` (a list of `{phase, effort?, model?}` entries) when committed, or `phase_signals_partial: true` when the operator force-stops. `brief_version: 2.1` activates the **sequencing gate**: validator emits `BRIEF_V51_MISSING_SIGNALS` if a 2.1-versioned brief lacks both fields. Downstream commands surface a friendly hint pointing back to `/trekbrief` — enforcement is validator-only. Composition is documented prose in each downstream command's `## Composition rule (v5.1)` section: `brief.phase_signals[phase] > profile.phase_models[phase]`. The brief signal wins per-phase when present; the profile fills gaps. `effort == low` activates each command's existing `--quick`-equivalent code-path (`/trekexecute` low effort = one session at a time). High-effort behavior is deferred to v5.1.1 per brief Non-Goal.
|
||||
|
||||
**Research:** Foreground workflow (v2.4.0): Parse mode → Interview → Parallel research swarm (5 local + 4 external + 1 bridge, spawned from main context) → Follow-ups → Triangulation → Synthesis + brief → Stats. With `--project`, writes to `{dir}/research/NN-slug.md`.
|
||||
**Research:** Foreground workflow (v2.4.0): Parse mode → Interview → Parallel research swarm (5 local + 4 external + 1 bridge, spawned from main context) → Follow-ups → Triangulation → Synthesis + brief. With `--project`, writes to `{dir}/research/NN-slug.md`.
|
||||
|
||||
**Plan:** Foreground workflow (v2.4.0): Parse mode (validate brief input) → Codebase sizing → Brief review (`brief-reviewer`) → Parallel exploration (6-8 agents, spawned from main context) → Deep-dives → Synthesis (with architecture-note cross-reference if present) → Planning → Adversarial review (`plan-critic` + `scope-guardian`) → Present/refine → Handoff. With `--project`, writes to `{dir}/plan.md` and auto-detects `{dir}/architecture/overview.md` (produced by an opt-in upstream architect plugin if installed; not bundled).
|
||||
|
||||
|
|
@ -87,7 +84,7 @@ No code-level dependency between plugins — the contract is filesystem-level on
|
|||
|
||||
### Primitives per step (decision matrix)
|
||||
|
||||
Which native Claude Code primitive each pipeline step runs on today, and the alternatives that were considered and where they landed. Companion to *Delegate the engine, keep the policy* above; the decision record lives in `docs/T1-cc26-delegated-orchestration.md`. The orchestration runs inline in the session, not in a delegated orchestrator agent, because the swarm's results end in the main context anyway (the orchestrator reference docs are removed).
|
||||
Which native Claude Code primitive each pipeline step runs on today, and the alternatives that were considered and where they landed. Companion to *Delegate the engine, keep the policy* above. The orchestration runs inline in the session, not in a delegated orchestrator agent, because the swarm's results end in the main context anyway (the orchestrator reference docs are removed).
|
||||
|
||||
| Step | Primitive today (production) | Alternatives considered (status) |
|
||||
|---|---|---|
|
||||
|
|
@ -97,7 +94,7 @@ Which native Claude Code primitive each pipeline step runs on today, and the alt
|
|||
| **execute** | Inline step loop; multi-session via `git worktree` + session orders the user runs, wave by wave; deterministic manifest audit | CC `TaskCreate`/`TodoWrite` for progress/resume (insufficient — carries no step status / attempts / SHA / drift → own typed `progress.json` contract) |
|
||||
| **review** | Inline parallel reviewers (no cross-feed) → `review-coordinator` Judge | **Workflow** substrate for Phase 5–6 (bake-off POSITIVE: +4.4 % tokens / +54 % wall-time → shipped opt-in, since removed; wholesale substrate swap declined) |
|
||||
| **continue** | Inline reads `.session-state.local.json` → zero-confirm resume | CC `--resume` (transcript replay, not typed work-state → insufficient) |
|
||||
| **cross-cutting** | 8 hook scripts: `pre-bash` + `pre-write` guards, `pre-agent-cap` loop-bound enforcement, `post-bash` stats, `session-title`, `pre-`/`post-compact` flush, **`Stop`→OTEL** export | — |
|
||||
| **cross-cutting** | 6 hook scripts: `pre-bash` + `pre-write` guards, `pre-agent-cap` loop-bound enforcement, `session-title`, `pre-`/`post-compact` flush | — |
|
||||
|
||||
¹ MCP per research agent: `docs-researcher` → Microsoft Learn + Tavily · `community-`/`security-`/`contrarian-researcher` → Tavily (+ WebSearch/WebFetch). Graceful degradation when an MCP server is absent.
|
||||
|
||||
|
|
@ -123,26 +120,6 @@ Legacy paths (still work without `--project`):
|
|||
- Session orders: `.claude/trekplan-sessions/{slug}/orders.md` (`--decompose`) and `orders/` (`/trekexecute` Phase 2.6: one order per session, `wave.json` while a wave awaits its sessions, `merged-waves.json`, and each session's archived `wave-W-session-N.progress.json`)
|
||||
- Progress: `{plan-dir}/.trekexecute-progress-{slug}.json`
|
||||
|
||||
Stats:
|
||||
- Brief stats: `${CLAUDE_PLUGIN_DATA}/trekbrief-stats.jsonl`
|
||||
- Plan stats: `${CLAUDE_PLUGIN_DATA}/trekplan-stats.jsonl`
|
||||
- Exec stats: `${CLAUDE_PLUGIN_DATA}/trekexecute-stats.jsonl`
|
||||
- Research stats: `${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl`
|
||||
- Continue stats: `${CLAUDE_PLUGIN_DATA}/trekcontinue-stats.jsonl`
|
||||
|
||||
The Bash tool env carries no `CLAUDE_PLUGIN_DATA`. Brief, plan and execute
|
||||
stats are written by `lib/stats/stats-append.mjs`, which falls back to
|
||||
`~/.claude/plugins/data/voyage-ktg-plugin-marketplace/`, the directory
|
||||
`scripts/yardstick.mjs` reads. `/trekexecute`'s lifecycle events go through
|
||||
`lib/stats/event-emit.mjs`, which resolves the same directory. A failed write
|
||||
is reported, not skipped. The research, review and continue stats phases still
|
||||
name the variable in prose only.
|
||||
A brief record also carries `order_id`, `order_claimed_at` and `order_reason`:
|
||||
the coord-queue order the run is working on, added by the writer from the
|
||||
queue's claimed orders (`lib/stats/claimed-order.mjs`), never by the model.
|
||||
`scripts/yardstick.mjs` joins its `use` number on `order_id`; the three fields
|
||||
are never exported.
|
||||
|
||||
## Terminology
|
||||
|
||||
- **Task brief** — produced by `/trekbrief`. Declares intent, goal, and research plan. Drives planning.
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ Per-command flag tables, imported from `CLAUDE.md` via pointer.
|
|||
| `--quick` | Proposes `small` as the class (still said aloud, overridable, raised when the task proves bigger) → brief.md with research plan |
|
||||
| `--gates [true\|false\|closed\|open]` | (v3.4.0) Autonomy flag: bare, `true` or `closed` = on; `false` or `open` = off; any other next token is not its value. See `## Autonomy mode` in `docs/operations.md`. |
|
||||
| `--profile <name>` | (v4.1.0) Model profile: `economy` / `balanced` / `premium` / `fable` / `<custom>`. Sets `phase_models` for the brief phase. See `## Profile system` in `docs/operations.md`. |
|
||||
| `--approve <project-dir \| brief-file>` | (veikart steg 1) No interview: show the existing brief's `## Intent` + `## Goal` verbatim, ask the operator to approve, and on "Approve" stamp `intent_approved_hash` + `intent_approved_at`, emit `brief-approved`, and write the trekbrief stats record (`"mode": "approve"`). The remedy `/trekplan` names when it halts on `BRIEF_INTENT_NOT_APPROVED` / `BRIEF_INTENT_APPROVAL_STALE`. |
|
||||
| `--approve <project-dir \| brief-file>` | (veikart steg 1) No interview: show the existing brief's `## Intent` + `## Goal` verbatim, ask the operator to approve, and on "Approve" stamp `intent_approved_hash` + `intent_approved_at`. The remedy `/trekplan` names when it halts on `BRIEF_INTENT_NOT_APPROVED` / `BRIEF_INTENT_APPROVAL_STALE`. |
|
||||
|
||||
Always interactive. Phase 3 puts the task in a size class (`ceremony:`, K1; heaviest in doubt, said aloud, overridable, only moved up) and asks one round of questions within the class budget (`small` 4, `bounded` 8, `architectural` 12 per round), recommendation first, facts looked up instead of asked, a visible Deferred list; it also asks where the brief is stored (`artifact_store: git | local`). Phase 4 runs a `brief-reviewer` stop-gate with 1 / 2 / 3 review iterations by class. Phase 4h takes only an explicit "Approve" («høres bra ut» and an empty answer are refused by `lib/util/approval-log.mjs answer`), stamps the brief, appends the approval to the target repo's log (`approval-log.mjs append`, K2) and commits it (`lib/util/artifact-commit.mjs`: the log always, the brief when `git`). After that, asks the user to choose manual (print commands) or auto (Claude runs research + plan in foreground).
|
||||
|
||||
|
|
|
|||
|
|
@ -1,227 +0,0 @@
|
|||
# Observability — voyage v4.1
|
||||
|
||||
This document describes the *opt-in* OpenTelemetry / Prometheus export
|
||||
path added in v4.1. The default JSONL stats stream
|
||||
(`${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl`) remains unchanged — it is the
|
||||
canonical event log and continues to be written regardless of OTel mode.
|
||||
|
||||
## Overview
|
||||
|
||||
Voyage v4.0 wrote per-command stats to JSONL files only. Operators who
|
||||
wanted dashboards had to roll their own log-pipeline. v4.1 adds a Stop-hook
|
||||
called `hooks/scripts/otel-export.mjs` that, when activated via
|
||||
`VOYAGE_EXPORT_MODE`, transforms the JSONL records into either a
|
||||
Prometheus textfile or OTLP/HTTP push at session-end.
|
||||
|
||||
The hook is *additive*. With `VOYAGE_EXPORT_MODE=off` (default), the
|
||||
binary exits silently and no work is done — your existing JSONL workflow
|
||||
is untouched.
|
||||
|
||||
## Activating OTel export
|
||||
|
||||
Set `VOYAGE_EXPORT_MODE` in the shell before invoking any voyage command:
|
||||
|
||||
```bash
|
||||
# Default — no export, JSONL only
|
||||
unset VOYAGE_EXPORT_MODE # equivalent to VOYAGE_EXPORT_MODE=off
|
||||
|
||||
# Path A — Prometheus textfile (recommended for local dashboards)
|
||||
export VOYAGE_EXPORT_MODE=textfile
|
||||
export VOYAGE_TEXTFILE_DIR=/var/lib/node_exporter/textfile
|
||||
|
||||
# Path B — OTLP/HTTP push (recommended for centralized telemetry)
|
||||
export VOYAGE_EXPORT_MODE=otlp
|
||||
export VOYAGE_OTEL_ENDPOINT=https://otel.example.com/v1/metrics
|
||||
```
|
||||
|
||||
`hooks/hooks.json` wires the Stop event to `otel-export.mjs`, so the
|
||||
export runs automatically when Claude Code finishes a session. No manual
|
||||
invocation is required.
|
||||
|
||||
## Output formats
|
||||
|
||||
| Mode | Wire format | Endpoint shape | Cardinality cap |
|
||||
|------|-------------|----------------|-----------------|
|
||||
| `textfile` | Prometheus exposition format (text) | local file: `${VOYAGE_TEXTFILE_DIR}/voyage.prom` | low — voyage controls labels |
|
||||
| `otlp` | OTLP/JSON v1.0 metric ResourceMetrics | HTTPS POST: `${VOYAGE_OTEL_ENDPOINT}` | low — same allowlist as textfile |
|
||||
| `off` | (none) | — | — |
|
||||
|
||||
Both formats apply the **same field allowlist** — see
|
||||
`lib/exporters/field-allowlist.mjs` for the per-schema list. Fields not in
|
||||
the allowlist are dropped before export. This is a CWE-212 mitigation:
|
||||
operator-defined endpoints must never receive accidentally-leaked
|
||||
operator-private data (paths, prompts, brief content).
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `VOYAGE_EXPORT_MODE` | `off` | One of `off` / `textfile` / `otlp` |
|
||||
| `VOYAGE_TEXTFILE_DIR` | `${CLAUDE_PLUGIN_DATA}` | Directory for `voyage.prom` (textfile mode) |
|
||||
| `VOYAGE_OTEL_ENDPOINT` | _(none)_ | HTTPS URL for OTLP/HTTP POST |
|
||||
| `VOYAGE_OTEL_ALLOW_PRIVATE` | _(unset)_ | Set to `1` to allow loopback / RFC1918 endpoints |
|
||||
| `VOYAGE_TOKEN_METER` | _(unset)_ | Set to a truthy value to capture per-session token/cost into `token-usage-stats.jsonl` on Stop (default off → zero added latency). See **Token/cost metering** below. |
|
||||
|
||||
## Docker Compose quickstart
|
||||
|
||||
A pre-pinned local stack lives at `examples/observability/`:
|
||||
|
||||
```bash
|
||||
cd examples/observability
|
||||
mkdir -p voyage-textfile
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
This brings up Prometheus, Grafana, node-exporter (textfile mode), and
|
||||
otel-collector (OTLP mode) on `localhost`. See
|
||||
`examples/observability/README.md` for endpoint URLs and version pins.
|
||||
|
||||
## Stats schema
|
||||
|
||||
Each Voyage command emits one JSONL record per significant event. Schemas
|
||||
are documented in `tests/fixtures/jsonl-schemas.md` (Step 1 of v4.1) and
|
||||
locked by `tests/lib/profile-stats-fields.test.mjs`.
|
||||
|
||||
The exporter applies the field allowlist defined in
|
||||
`lib/exporters/field-allowlist.mjs`. Adding a new field to the JSONL
|
||||
schema does **not** automatically expose it in OTel — you must add it to
|
||||
the allowlist explicitly. This is intentional: `${CLAUDE_PLUGIN_DATA}` is
|
||||
trusted local storage; OTel endpoints are operator-controlled and may be
|
||||
external.
|
||||
|
||||
## Token/cost metering
|
||||
|
||||
> **SKAL-2.** Opt-in capture of per-session token usage and a cache-aware USD
|
||||
> cost estimate, folded into the existing Stop hook (`hooks/scripts/otel-export.mjs`).
|
||||
> No new hook is added — capture rides the same Stop event, so there is no
|
||||
> second-Stop-hook ordering race.
|
||||
|
||||
**Activation.** Set `VOYAGE_TOKEN_METER` to any truthy value. When unset (the
|
||||
default) the capture path is skipped entirely — zero added Stop latency. When
|
||||
set, the hook reads the Claude Code transcript (`transcript_path` from the Stop
|
||||
payload), sums token usage, derives cost, and **upserts** one record per session
|
||||
into `${CLAUDE_PLUGIN_DATA}/token-usage-stats.jsonl`. Capture is fail-open: any
|
||||
error (malformed payload, unreadable transcript) is swallowed and never blocks
|
||||
Stop. Once captured, the record is exported like any other stats file when
|
||||
`VOYAGE_EXPORT_MODE` is `textfile` or `otlp`.
|
||||
|
||||
**Schema (`token-usage`).** Flat numeric record:
|
||||
`ts`, `session_id`, `scope`, `model`, `tokens_input`, `tokens_output`,
|
||||
`tokens_cache_creation`, `tokens_cache_read`, `cost_usd`, `is_estimate`,
|
||||
`price_table_version`. The field allowlist (`lib/exporters/field-allowlist.mjs`,
|
||||
`TOKEN_USAGE_ALLOWED`) admits only the numeric + low-cardinality-label fields and
|
||||
**strips `session_id` at export** (CWE-212). The exporter auto-promotes each
|
||||
numeric field to a metric: `voyage_token_usage_tokens_input` (Prometheus) /
|
||||
`voyage.token-usage.tokens_input` (OTLP).
|
||||
|
||||
**Cost contract (honesty).** `cost_usd` is computed from a dated, in-source
|
||||
`PRICE_TABLE` (per-MTok USD), cache-aware:
|
||||
`input + output + cache_creation×(5m write rate) + cache_read×(read rate)`.
|
||||
Each record carries `price_table_version` (the date the prices were resolved)
|
||||
and `is_estimate`. When the transcript's model is **not** in the price table, the
|
||||
meter **refuses to guess**: `cost_usd` is `null` and `is_estimate` is `true`.
|
||||
Prices are volatile — re-resolve them against the `claude-api` reference and bump
|
||||
`PRICE_TABLE_VERSION` when they change.
|
||||
|
||||
**v1 limitation — MAIN-CONTEXT only.** The meter reads the main-session
|
||||
transcript, which contains only `isSidechain:false` records. Sub-agent (swarm)
|
||||
turns live in separate `agent-*.jsonl` sibling files and are **not** counted, so
|
||||
`cost_usd` is a lower bound on total Voyage cost, not the session total. Every
|
||||
record is stamped `scope:'main-context'` to make this explicit. Per-subagent
|
||||
attribution is a documented v2 follow-on (it was a Non-Goal for v1).
|
||||
|
||||
## Security
|
||||
|
||||
The exporter is hardened against three CWE classes:
|
||||
|
||||
- **CWE-22 (path traversal)** — `lib/exporters/path-validator.mjs`
|
||||
rejects relative paths, symlinks, and paths outside `allowedRoots`
|
||||
(`VOYAGE_TEXTFILE_DIR` and `CLAUDE_PLUGIN_DATA`). Tested in
|
||||
`tests/hooks/otel-export-validators.test.mjs`.
|
||||
- **CWE-918 (SSRF)** — `lib/exporters/endpoint-validator.mjs` requires
|
||||
HTTPS, blocks loopback (127.0.0.0/8) and RFC1918 (10/8, 172.16/12,
|
||||
192.168/16), unless `VOYAGE_OTEL_ALLOW_PRIVATE=1` is set explicitly.
|
||||
Cloud metadata endpoints (169.254.169.254) are permanently blocked.
|
||||
- **CWE-212 (improper data sanitization)** — every record passes through
|
||||
`lib/exporters/field-allowlist.mjs` before any I/O. Adding a field to
|
||||
the JSONL stream does not expose it externally; operators must update
|
||||
the allowlist intentionally.
|
||||
|
||||
### Minimum versions per CVE history
|
||||
|
||||
| Component | Minimum version | Reason |
|
||||
|-----------|-----------------|--------|
|
||||
| `otel/opentelemetry-collector-contrib` | `0.115.0` | post-CVE-2024-42368 |
|
||||
| `prom/prometheus` | `3.0.1` | OOM regression fix in 2.x |
|
||||
| `prom/node-exporter` | `1.10.2` | textfile collector path normalization |
|
||||
| `grafana/grafana` | `11.4.0` | datasource provisioning hardening |
|
||||
|
||||
## Why direct export rather than a native collector
|
||||
|
||||
A balance review (V32) asked
|
||||
whether the custom exporters should be dropped in favour of pointing the
|
||||
standard `OTEL_*` environment variables at a co-located OTLP collector,
|
||||
letting that collector own egress and field selection. The operator
|
||||
decision (D2, 2026-06-20) is to **keep direct export**. The rationale is
|
||||
the security boundary, not a preference for re-hosting a collector:
|
||||
|
||||
- **The three guards run in-process, before any byte leaves Voyage.**
|
||||
`path-validator.mjs` (CWE-22), `endpoint-validator.mjs` (CWE-918 / SSRF),
|
||||
and `field-allowlist.mjs` (CWE-212) are applied inside `otel-export.mjs`
|
||||
and covered by `tests/hooks/otel-export-validators.test.mjs`. The
|
||||
records carry operator-private data (paths, prompts, brief content);
|
||||
the allowlist drops everything not explicitly named before export.
|
||||
- **A native-collector design moves that boundary out of audited code.**
|
||||
Handing raw JSONL to a sidecar collector means either re-expressing the
|
||||
field allowlist in collector config (a second source of truth that can
|
||||
drift) or shipping un-allowlisted private fields and trusting the
|
||||
collector's egress rules. The S21 SSRF hardening — 169.254.169.254
|
||||
permanently blocked, loopback/RFC1918 gated behind
|
||||
`VOYAGE_OTEL_ALLOW_PRIVATE` — is a property of `endpoint-validator.mjs`
|
||||
and would have to be re-created in collector configuration to be
|
||||
preserved.
|
||||
- **The collector path is still available, by design.** Operators who
|
||||
want collector semantics (retry, persistence, relabelling) use
|
||||
`textfile` mode and scrape `voyage.prom` with node-exporter / vector /
|
||||
otel-collector. Direct export is the minimal default, not a rejection
|
||||
of collectors — it keeps the data-sanitization boundary in Voyage's
|
||||
own validated code for the common case.
|
||||
|
||||
This is a deliberate direct-export-over-collector choice; the custom
|
||||
exporters and their guards are kept, not pruned.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Stop-hook is normal-exit only.** If Claude Code crashes or is killed
|
||||
with SIGKILL, the final session's metrics are not flushed. Use
|
||||
`--resume` on next start to recover plan/progress state; the missing
|
||||
session will not appear in dashboards.
|
||||
- **Tail-latency NFR is best-effort.** Textfile mode targets <5 ms p99,
|
||||
OTLP <1500 ms (AbortController guards). If the network endpoint is
|
||||
slow, the timeout fires and stats for that session are dropped — the
|
||||
hook always exits 0 to avoid blocking session shutdown.
|
||||
- **No retry on transport failure.** Stop-hook runs at most once per
|
||||
session. If the OTLP endpoint is unreachable, that session's metrics
|
||||
are lost. Production deployments should use `textfile` + a robust
|
||||
scrape pipeline (node-exporter, vector, otel-collector with persistent
|
||||
queue) to handle delivery semantics.
|
||||
- **No per-tenant labelling.** v4.1 emits flat metrics with command and
|
||||
schema_id labels only. Multi-tenant deployments needing per-user or
|
||||
per-project segmentation should layer a relabel stage in their
|
||||
collector or use external metadata.
|
||||
|
||||
## Cost-estimering disclaimer
|
||||
|
||||
The ROUGE-L / Jaccard / character n-gram thresholds in v4.1 are
|
||||
*starting points*, not contractual SLAs. The brief Risk-tabell explicitly
|
||||
flags these as anslag — they were calibrated against synthetic plan
|
||||
runs (Step 17) using `economy` and `premium` profiles. Real cross-tier
|
||||
agreement varies by task complexity. Treat the thresholds as smoke-test
|
||||
floors; tighten them in v4.2 once you have ≥10 production runs of data.
|
||||
|
||||
## See also
|
||||
|
||||
- `examples/observability/` — local Docker Compose stack
|
||||
- `tests/fixtures/jsonl-schemas.md` — canonical record shapes
|
||||
- `lib/exporters/field-allowlist.mjs` — per-schema allowed fields
|
||||
- `hooks/scripts/otel-export.mjs` — Stop-hook orchestrator
|
||||
|
|
@ -6,7 +6,7 @@ Imported from `CLAUDE.md` via pointer.
|
|||
|
||||
All four pipeline commands accept `--gates [true|false|closed|open]`. A bare `--gates`, `--gates true` or `--gates closed` turns gating **on**; `--gates false`, `--gates open` or no flag (the default) leaves it off. The next token is the flag's value only when it is one of those four; any other token stays the command's own argument, so `/trekplan --gates --project <dir>` and `/trekexecute --gates plan.md` are a bare `--gates`. Each command maps the value once through `lib/util/autonomy-gate.mjs`; a shim that fails falls back to the same mapping in prose, and the run goes on. With gating on, `/trekbrief` pauses before its auto chain, `/trekresearch` after each topic, and `/trekplan` after exploration, synthesis and adversarial review. `/trekexecute` accepts the flag and pauses on nothing: its one stop is the merge into main, which always asks (below).
|
||||
|
||||
Under the hood: `lib/util/autonomy-gate.mjs` is the `--gates` flag parser. It prints `{"ok":true,"gates":true}` when `--gates` is bare, `true` or `closed`, `{"ok":true,"gates":false}` for `false`, `open` or no flag, and a usage error (exit 1) for anything else. The module is pure data with no I/O. Separately, `lib/stats/event-emit.mjs` records named lifecycle events (`brief-approved`, `main-merge-gate`, `main-merge-approved`, `main-merge-declined`, `user_input`) to `trekexecute-stats.jsonl` in the plugin data directory (`CLAUDE_PLUGIN_DATA`, else `~/.claude/plugins/data/voyage-ktg-plugin-marketplace/`, the directory `scripts/yardstick.mjs` reads); a write that fails is reported, never skipped. The main-merge gate is the final autonomy boundary before HEAD lands on `main`.
|
||||
Under the hood: `lib/util/autonomy-gate.mjs` is the `--gates` flag parser. It prints `{"ok":true,"gates":true}` when `--gates` is bare, `true` or `closed`, `{"ok":true,"gates":false}` for `false`, `open` or no flag, and a usage error (exit 1) for anything else. The module is pure data with no I/O. The main-merge gate is the final autonomy boundary before HEAD lands on `main`.
|
||||
|
||||
**The main-merge gate is a lock, and the guard asks the one question** (operator decision 2026-10-08). The gate opens only when the run is `completed` and the review's verdict gate (`lib/review/verdict-gate.mjs`, K7) exits 0; BLOCK (1) and "could not decide" (2) keep it closed. When it opens, `/trekexecute` runs the merge into main itself and `pre-bash-executor` answers "ask" before it runs, in auto mode too, so that question is the only one. In the permission mode where Claude Code shows no prompts, the guard refuses the merge instead, and every other call it would ask about (push, tag or publish, destructive git), since no one can be asked (operator decisions 2026-10-09). The executor no longer asks "ready to merge?" first, and `--gates` no longer adds a pause before the wave merge-back: the guard asks there as well, once per session, because the push and the merge go in one call. Before this, one merge cost two questions in three places: the `--gates` pause before the wave merge-back, the gate's own question after the waves were merged, and the gate's question on the single-session path followed by the guard's at the merge itself.
|
||||
|
||||
|
|
@ -16,7 +16,7 @@ Three architectural options were considered for the speedup work:
|
|||
|
||||
- **Path A — cache-first** (drop `--allowedTools` per child to recover cross-phase cache sharing): REJECTED. Inverts the security model; plugin hooks don't fire reliably in `claude -p` (research/06 GH #36071).
|
||||
- **Path B — sequential `--no-ff` parallel waves with manifest-driven failure recovery**: CHOSEN. Ships in v3.4.0. Phase 2.6 of `/trekexecute` runs the wave executor with hardenings for plugin-in-monorepo + gitignored-state topology.
|
||||
- **Path C — hybrid (cache-warm sentinel + identical-tool parallel)**: **CLOSED 2026-05-05.** Q3 experiment measured median `cache_creation_input_tokens` = 163,903 across 3 fork-children at 186K parent context (CC v2.1.128, Sonnet 4.6). Master-plan thresholds: ≤ 1,500 POSITIVE / ≥ 3,500 NEGATIVE. Result is solidly NEGATIVE — `CLAUDE_CODE_FORK_SUBAGENT` does not preserve cache prefix across identical-tool children at our context size. Path C migration is deferred indefinitely; reassessment is appropriate when CC v2.2.xxx ships fork-cache-relevant features. Companion analyser: `lib/stats/cache-analyzer.mjs`.
|
||||
- **Path C — hybrid (cache-warm sentinel + identical-tool parallel)**: **CLOSED 2026-05-05.** Q3 experiment measured median `cache_creation_input_tokens` = 163,903 across 3 fork-children at 186K parent context (CC v2.1.128, Sonnet 4.6). Master-plan thresholds: ≤ 1,500 POSITIVE / ≥ 3,500 NEGATIVE. Result is solidly NEGATIVE — `CLAUDE_CODE_FORK_SUBAGENT` does not preserve cache prefix across identical-tool children at our context size. Path C migration is deferred indefinitely; reassessment is appropriate when CC v2.2.xxx ships fork-cache-relevant features.
|
||||
|
||||
> **Status (6.0.0):** `/trekexecute` no longer starts wave children. Phase 2.6 prepares one wave at a time and hands each session over as an order the user runs in a session they open (CHANGELOG, 6.0.0). The Q3 harness was removed in 6.0.0 because it started the binary; its result above stands, and the script remains in git history before that tag.
|
||||
|
||||
|
|
@ -46,16 +46,6 @@ Create `voyage-profiles/<custom>.yaml` in the repo root (or `~/.claude/voyage-pr
|
|||
|
||||
Drift between plan-frontmatter `profile:` and step-manifest `profile_used:` emits a `MANIFEST_PROFILE_DRIFT` warning from `plan-validator --strict` (Step 20). Plan remains valid; the warning surfaces accidental tier-mismatch.
|
||||
|
||||
## Observability (Stop hook, v4.1.0)
|
||||
## Observability
|
||||
|
||||
The `Stop` hook in `hooks/hooks.json` runs `hooks/scripts/otel-export.mjs` at session-end. The hook is **opt-in** — when `VOYAGE_EXPORT_MODE` is unset or `off`, no work is done.
|
||||
|
||||
| Mode | Output | Endpoint env-var |
|
||||
|------|--------|------------------|
|
||||
| `off` (default) | _(no export)_ | — |
|
||||
| `textfile` | `voyage.prom` (Prometheus exposition format) | `VOYAGE_TEXTFILE_DIR` |
|
||||
| `otlp` | OTLP/JSON POST | `VOYAGE_OTEL_ENDPOINT` |
|
||||
|
||||
Endpoint validation: `VOYAGE_OTEL_ALLOW_PRIVATE=1` is required to send to loopback or RFC1918 destinations (CWE-918 SSRF mitigation). Allowlist `lib/exporters/field-allowlist.mjs` redacts records before export (CWE-212). Path validation (`lib/exporters/path-validator.mjs`) rejects symlink + traversal (CWE-22).
|
||||
|
||||
Local Docker Compose stack: `examples/observability/`. Operator docs: `docs/observability.md`. Both pin minimum versions per CVE history (`prom/prometheus:v3.0.1`, `grafana/grafana:11.4.0`, `otel/opentelemetry-collector-contrib:0.115.0`).
|
||||
Claude Code's own OpenTelemetry (`CLAUDE_CODE_ENABLE_TELEMETRY`) gives tool durations, tokens and cost per agent and skill: see [Monitoring usage](https://code.claude.com/docs/en/monitoring-usage) (read 2026-11-07, Claude Code 2.1.319).
|
||||
|
|
|
|||
|
|
@ -133,7 +133,7 @@ same trace.
|
|||
|
||||
| Change | Why |
|
||||
|--------|-----|
|
||||
| The case-4 scaffold writes a current `intent_approved_hash` into the brief | The intent gate now passes, so only the sequencing gate can stop the case. The marker is written literally, not by `intent-approval.mjs --stamp`: `--stamp` emits a `brief-approved` record into the live stats that the yardstick reads |
|
||||
| The case-4 scaffold writes a current `intent_approved_hash` into the brief | The intent gate now passes, so only the sequencing gate can stop the case. The marker is written literally, not by `intent-approval.mjs --stamp` |
|
||||
| New case-4 grader `no-intent-halt` (trace, `not_contains`, any `BRIEF_INTENT_*` code) | If the marker ever goes stale, the case fails and names the reason. A halt on the intent gate no longer counts as a pass |
|
||||
| New case-4 grader `names-code` (the validator's `BRIEF_V51_MISSING_SIGNALS` code in the trace) | The known-positive below: without it, the case passed with the gate's code switched off |
|
||||
| `says-fail` / `says-pass` take `FAILED` / `PASSED` as the verdict word | v2's word boundary rejected them. v1's raw substring accepted them. A correct reply must not fail the case |
|
||||
|
|
|
|||
|
|
@ -229,7 +229,6 @@ numbers after ≥ 3 runs of each profile.
|
|||
|
||||
- [`README.md` § Profile system](../README.md) — top-level overview
|
||||
- [`CLAUDE.md` § Profile system](../CLAUDE.md) — internal reference
|
||||
- [`docs/observability.md`](observability.md) — JSONL → OTel pipeline
|
||||
- [`tests/synthetic/profile-jaccard-calibration.md`](../tests/synthetic/profile-jaccard-calibration.md)
|
||||
— calibration status and threshold rationale
|
||||
- [`lib/profiles/`](../lib/profiles/) — built-in profile YAMLs
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ By stage — Plan 0/1/0 · Design 0/0/1 · Build 2/3/1 · Test 0/2/0 · Deploy 0
|
|||
| P11 | AI in the PR review loop (Deploy) | partial | `commands/trekreview.md`; `lib/review/rule-catalogue.mjs`; `lib/review/coordinator-contract.mjs` | Independent reviewers, a judge, a versioned rule catalogue and a four-level severity taxonomy are strongly implemented — but the input is a SHA-range diff, not a PR: no forge integration, no code-owner approval, and the verdict blocks nothing |
|
||||
| P12 | Hooks as approval gates (Deploy) | partial | `hooks/scripts/pre-bash-executor.mjs` (asks before merge into main and push); `commands/trekexecute.md` main-merge gate | The guard answers "ask" before merge into main and push, in auto mode too, and the main-merge gate is a lock on the review verdict — but the other approval pauses are `AskUserQuestion` prose in command files |
|
||||
| P13 | CI/CD integration and deployment (Deploy) | none | — | No CI workflows, no deploy/status/rollback tooling, no per-environment autonomy ladder |
|
||||
| P14 | Close the loop from maintenance (Maintain) | none | `hooks/scripts/otel-export.mjs`, `lib/exporters/`, `lib/stats/` | Observability exists but measures Voyage's own runs, not a product in production; no control bands, and the one feedback path (review → plan) is started by a human |
|
||||
| P14 | Close the loop from maintenance (Maintain) | none | — | No control bands, and the one feedback path (review → plan) is started by a human |
|
||||
| P15 | Recurring codebase scans (Maintain) | none | `hooks/scripts/pre-bash-executor.mjs` | No scheduler and no periodic entry point; the only `cron` hits are the rule that blocks crontab persistence |
|
||||
| P16 | Agent on call in chat (Maintain) | none | `README.md` | Declared an intentional omission for a solo project |
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue