227 lines
11 KiB
Markdown
227 lines
11 KiB
Markdown
# Observability — voyage v4.1
|
||
|
||
This document describes the *opt-in* OpenTelemetry / Prometheus export
|
||
path added in v4.1. The default JSONL stats stream
|
||
(`${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl`) remains unchanged — it is the
|
||
canonical event log and continues to be written regardless of OTel mode.
|
||
|
||
## Overview
|
||
|
||
Voyage v4.0 wrote per-command stats to JSONL files only. Operators who
|
||
wanted dashboards had to roll their own log-pipeline. v4.1 adds a Stop-hook
|
||
called `hooks/scripts/otel-export.mjs` that, when activated via
|
||
`VOYAGE_EXPORT_MODE`, transforms the JSONL records into either a
|
||
Prometheus textfile or OTLP/HTTP push at session-end.
|
||
|
||
The hook is *additive*. With `VOYAGE_EXPORT_MODE=off` (default), the
|
||
binary exits silently and no work is done — your existing JSONL workflow
|
||
is untouched.
|
||
|
||
## Activating OTel export
|
||
|
||
Set `VOYAGE_EXPORT_MODE` in the shell before invoking any voyage command:
|
||
|
||
```bash
|
||
# Default — no export, JSONL only
|
||
unset VOYAGE_EXPORT_MODE # equivalent to VOYAGE_EXPORT_MODE=off
|
||
|
||
# Path A — Prometheus textfile (recommended for local dashboards)
|
||
export VOYAGE_EXPORT_MODE=textfile
|
||
export VOYAGE_TEXTFILE_DIR=/var/lib/node_exporter/textfile
|
||
|
||
# Path B — OTLP/HTTP push (recommended for centralized telemetry)
|
||
export VOYAGE_EXPORT_MODE=otlp
|
||
export VOYAGE_OTEL_ENDPOINT=https://otel.example.com/v1/metrics
|
||
```
|
||
|
||
`hooks/hooks.json` wires the Stop event to `otel-export.mjs`, so the
|
||
export runs automatically when Claude Code finishes a session. No manual
|
||
invocation is required.
|
||
|
||
## Output formats
|
||
|
||
| Mode | Wire format | Endpoint shape | Cardinality cap |
|
||
|------|-------------|----------------|-----------------|
|
||
| `textfile` | Prometheus exposition format (text) | local file: `${VOYAGE_TEXTFILE_DIR}/voyage.prom` | low — voyage controls labels |
|
||
| `otlp` | OTLP/JSON v1.0 metric ResourceMetrics | HTTPS POST: `${VOYAGE_OTEL_ENDPOINT}` | low — same allowlist as textfile |
|
||
| `off` | (none) | — | — |
|
||
|
||
Both formats apply the **same field allowlist** — see
|
||
`lib/exporters/field-allowlist.mjs` for the per-schema list. Fields not in
|
||
the allowlist are dropped before export. This is a CWE-212 mitigation:
|
||
operator-defined endpoints must never receive accidentally-leaked
|
||
operator-private data (paths, prompts, brief content).
|
||
|
||
## Environment variables
|
||
|
||
| Variable | Default | Purpose |
|
||
|----------|---------|---------|
|
||
| `VOYAGE_EXPORT_MODE` | `off` | One of `off` / `textfile` / `otlp` |
|
||
| `VOYAGE_TEXTFILE_DIR` | `${CLAUDE_PLUGIN_DATA}` | Directory for `voyage.prom` (textfile mode) |
|
||
| `VOYAGE_OTEL_ENDPOINT` | _(none)_ | HTTPS URL for OTLP/HTTP POST |
|
||
| `VOYAGE_OTEL_ALLOW_PRIVATE` | _(unset)_ | Set to `1` to allow loopback / RFC1918 endpoints |
|
||
| `VOYAGE_TOKEN_METER` | _(unset)_ | Set to a truthy value to capture per-session token/cost into `token-usage-stats.jsonl` on Stop (default off → zero added latency). See **Token/cost metering** below. |
|
||
|
||
## Docker Compose quickstart
|
||
|
||
A pre-pinned local stack lives at `examples/observability/`:
|
||
|
||
```bash
|
||
cd examples/observability
|
||
mkdir -p voyage-textfile
|
||
docker compose up -d
|
||
```
|
||
|
||
This brings up Prometheus, Grafana, node-exporter (textfile mode), and
|
||
otel-collector (OTLP mode) on `localhost`. See
|
||
`examples/observability/README.md` for endpoint URLs and version pins.
|
||
|
||
## Stats schema
|
||
|
||
Each Voyage command emits one JSONL record per significant event. Schemas
|
||
are documented in `tests/fixtures/jsonl-schemas.md` (Step 1 of v4.1) and
|
||
locked by `tests/lib/profile-stats-fields.test.mjs`.
|
||
|
||
The exporter applies the field allowlist defined in
|
||
`lib/exporters/field-allowlist.mjs`. Adding a new field to the JSONL
|
||
schema does **not** automatically expose it in OTel — you must add it to
|
||
the allowlist explicitly. This is intentional: `${CLAUDE_PLUGIN_DATA}` is
|
||
trusted local storage; OTel endpoints are operator-controlled and may be
|
||
external.
|
||
|
||
## Token/cost metering
|
||
|
||
> **SKAL-2.** Opt-in capture of per-session token usage and a cache-aware USD
|
||
> cost estimate, folded into the existing Stop hook (`hooks/scripts/otel-export.mjs`).
|
||
> No new hook is added — capture rides the same Stop event, so there is no
|
||
> second-Stop-hook ordering race.
|
||
|
||
**Activation.** Set `VOYAGE_TOKEN_METER` to any truthy value. When unset (the
|
||
default) the capture path is skipped entirely — zero added Stop latency. When
|
||
set, the hook reads the Claude Code transcript (`transcript_path` from the Stop
|
||
payload), sums token usage, derives cost, and **upserts** one record per session
|
||
into `${CLAUDE_PLUGIN_DATA}/token-usage-stats.jsonl`. Capture is fail-open: any
|
||
error (malformed payload, unreadable transcript) is swallowed and never blocks
|
||
Stop. Once captured, the record is exported like any other stats file when
|
||
`VOYAGE_EXPORT_MODE` is `textfile` or `otlp`.
|
||
|
||
**Schema (`token-usage`).** Flat numeric record:
|
||
`ts`, `session_id`, `scope`, `model`, `tokens_input`, `tokens_output`,
|
||
`tokens_cache_creation`, `tokens_cache_read`, `cost_usd`, `is_estimate`,
|
||
`price_table_version`. The field allowlist (`lib/exporters/field-allowlist.mjs`,
|
||
`TOKEN_USAGE_ALLOWED`) admits only the numeric + low-cardinality-label fields and
|
||
**strips `session_id` at export** (CWE-212). The exporter auto-promotes each
|
||
numeric field to a metric: `voyage_token_usage_tokens_input` (Prometheus) /
|
||
`voyage.token-usage.tokens_input` (OTLP).
|
||
|
||
**Cost contract (honesty).** `cost_usd` is computed from a dated, in-source
|
||
`PRICE_TABLE` (per-MTok USD), cache-aware:
|
||
`input + output + cache_creation×(5m write rate) + cache_read×(read rate)`.
|
||
Each record carries `price_table_version` (the date the prices were resolved)
|
||
and `is_estimate`. When the transcript's model is **not** in the price table, the
|
||
meter **refuses to guess**: `cost_usd` is `null` and `is_estimate` is `true`.
|
||
Prices are volatile — re-resolve them against the `claude-api` reference and bump
|
||
`PRICE_TABLE_VERSION` when they change.
|
||
|
||
**v1 limitation — MAIN-CONTEXT only.** The meter reads the main-session
|
||
transcript, which contains only `isSidechain:false` records. Sub-agent (swarm)
|
||
turns live in separate `agent-*.jsonl` sibling files and are **not** counted, so
|
||
`cost_usd` is a lower bound on total Voyage cost, not the session total. Every
|
||
record is stamped `scope:'main-context'` to make this explicit. Per-subagent
|
||
attribution is a documented v2 follow-on (it was a Non-Goal for v1).
|
||
|
||
## Security
|
||
|
||
The exporter is hardened against three CWE classes:
|
||
|
||
- **CWE-22 (path traversal)** — `lib/exporters/path-validator.mjs`
|
||
rejects relative paths, symlinks, and paths outside `allowedRoots`
|
||
(`VOYAGE_TEXTFILE_DIR` and `CLAUDE_PLUGIN_DATA`). Tested in
|
||
`tests/hooks/otel-export-validators.test.mjs`.
|
||
- **CWE-918 (SSRF)** — `lib/exporters/endpoint-validator.mjs` requires
|
||
HTTPS, blocks loopback (127.0.0.0/8) and RFC1918 (10/8, 172.16/12,
|
||
192.168/16), unless `VOYAGE_OTEL_ALLOW_PRIVATE=1` is set explicitly.
|
||
Cloud metadata endpoints (169.254.169.254) are permanently blocked.
|
||
- **CWE-212 (improper data sanitization)** — every record passes through
|
||
`lib/exporters/field-allowlist.mjs` before any I/O. Adding a field to
|
||
the JSONL stream does not expose it externally; operators must update
|
||
the allowlist intentionally.
|
||
|
||
### Minimum versions per CVE history
|
||
|
||
| Component | Minimum version | Reason |
|
||
|-----------|-----------------|--------|
|
||
| `otel/opentelemetry-collector-contrib` | `0.115.0` | post-CVE-2024-42368 |
|
||
| `prom/prometheus` | `3.0.1` | OOM regression fix in 2.x |
|
||
| `prom/node-exporter` | `1.10.2` | textfile collector path normalization |
|
||
| `grafana/grafana` | `11.4.0` | datasource provisioning hardening |
|
||
|
||
## Why direct export rather than a native collector
|
||
|
||
A balance review (`docs/voyage-vs-cc-balance-analysis.md` §4, V32) asked
|
||
whether the custom exporters should be dropped in favour of pointing the
|
||
standard `OTEL_*` environment variables at a co-located OTLP collector,
|
||
letting that collector own egress and field selection. The operator
|
||
decision (D2, 2026-06-20) is to **keep direct export**. The rationale is
|
||
the security boundary, not a preference for re-hosting a collector:
|
||
|
||
- **The three guards run in-process, before any byte leaves Voyage.**
|
||
`path-validator.mjs` (CWE-22), `endpoint-validator.mjs` (CWE-918 / SSRF),
|
||
and `field-allowlist.mjs` (CWE-212) are applied inside `otel-export.mjs`
|
||
and covered by `tests/hooks/otel-export-validators.test.mjs`. The
|
||
records carry operator-private data (paths, prompts, brief content);
|
||
the allowlist drops everything not explicitly named before export.
|
||
- **A native-collector design moves that boundary out of audited code.**
|
||
Handing raw JSONL to a sidecar collector means either re-expressing the
|
||
field allowlist in collector config (a second source of truth that can
|
||
drift) or shipping un-allowlisted private fields and trusting the
|
||
collector's egress rules. The S21 SSRF hardening — 169.254.169.254
|
||
permanently blocked, loopback/RFC1918 gated behind
|
||
`VOYAGE_OTEL_ALLOW_PRIVATE` — is a property of `endpoint-validator.mjs`
|
||
and would have to be re-created in collector configuration to be
|
||
preserved.
|
||
- **The collector path is still available, by design.** Operators who
|
||
want collector semantics (retry, persistence, relabelling) use
|
||
`textfile` mode and scrape `voyage.prom` with node-exporter / vector /
|
||
otel-collector. Direct export is the minimal default, not a rejection
|
||
of collectors — it keeps the data-sanitization boundary in Voyage's
|
||
own validated code for the common case.
|
||
|
||
This is a deliberate direct-export-over-collector choice; the custom
|
||
exporters and their guards are kept, not pruned.
|
||
|
||
## Limitations
|
||
|
||
- **Stop-hook is normal-exit only.** If Claude Code crashes or is killed
|
||
with SIGKILL, the final session's metrics are not flushed. Use
|
||
`--resume` on next start to recover plan/progress state; the missing
|
||
session will not appear in dashboards.
|
||
- **Tail-latency NFR is best-effort.** Textfile mode targets <5 ms p99,
|
||
OTLP <1500 ms (AbortController guards). If the network endpoint is
|
||
slow, the timeout fires and stats for that session are dropped — the
|
||
hook always exits 0 to avoid blocking session shutdown.
|
||
- **No retry on transport failure.** Stop-hook runs at most once per
|
||
session. If the OTLP endpoint is unreachable, that session's metrics
|
||
are lost. Production deployments should use `textfile` + a robust
|
||
scrape pipeline (node-exporter, vector, otel-collector with persistent
|
||
queue) to handle delivery semantics.
|
||
- **No per-tenant labelling.** v4.1 emits flat metrics with command and
|
||
schema_id labels only. Multi-tenant deployments needing per-user or
|
||
per-project segmentation should layer a relabel stage in their
|
||
collector or use external metadata.
|
||
|
||
## Cost-estimering disclaimer
|
||
|
||
The ROUGE-L / Jaccard / character n-gram thresholds in v4.1 are
|
||
*starting points*, not contractual SLAs. The brief Risk-tabell explicitly
|
||
flags these as anslag — they were calibrated against synthetic plan
|
||
runs (Step 17) using `economy` and `premium` profiles. Real cross-tier
|
||
agreement varies by task complexity. Treat the thresholds as smoke-test
|
||
floors; tighten them in v4.2 once you have ≥10 production runs of data.
|
||
|
||
## See also
|
||
|
||
- `examples/observability/` — local Docker Compose stack
|
||
- `tests/fixtures/jsonl-schemas.md` — canonical record shapes
|
||
- `lib/exporters/field-allowlist.mjs` — per-schema allowed fields
|
||
- `hooks/scripts/otel-export.mjs` — Stop-hook orchestrator
|