Kjell Tore Guttormsen 7e78d076f9 docs(observability): document token-usage schema + main-context v1 scope

2026-06-26 14:47:24 +02:00

11 KiB

Raw Blame History

Observability — voyage v4.1

This document describes the opt-in OpenTelemetry / Prometheus export path added in v4.1. The default JSONL stats stream (${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl) remains unchanged — it is the canonical event log and continues to be written regardless of OTel mode.

Overview

Voyage v4.0 wrote per-command stats to JSONL files only. Operators who wanted dashboards had to roll their own log-pipeline. v4.1 adds a Stop-hook called hooks/scripts/otel-export.mjs that, when activated via VOYAGE_EXPORT_MODE, transforms the JSONL records into either a Prometheus textfile or OTLP/HTTP push at session-end.

The hook is additive. With VOYAGE_EXPORT_MODE=off (default), the binary exits silently and no work is done — your existing JSONL workflow is untouched.

Activating OTel export

Set VOYAGE_EXPORT_MODE in the shell before invoking any voyage command:

# Default — no export, JSONL only
unset VOYAGE_EXPORT_MODE                  # equivalent to VOYAGE_EXPORT_MODE=off

# Path A — Prometheus textfile (recommended for local dashboards)
export VOYAGE_EXPORT_MODE=textfile
export VOYAGE_TEXTFILE_DIR=/var/lib/node_exporter/textfile

# Path B — OTLP/HTTP push (recommended for centralized telemetry)
export VOYAGE_EXPORT_MODE=otlp
export VOYAGE_OTEL_ENDPOINT=https://otel.example.com/v1/metrics

hooks/hooks.json wires the Stop event to otel-export.mjs, so the export runs automatically when Claude Code finishes a session. No manual invocation is required.

Output formats

Mode	Wire format	Endpoint shape	Cardinality cap
`textfile`	Prometheus exposition format (text)	local file: `${VOYAGE_TEXTFILE_DIR}/voyage.prom`	low — voyage controls labels
`otlp`	OTLP/JSON v1.0 metric ResourceMetrics	HTTPS POST: `${VOYAGE_OTEL_ENDPOINT}`	low — same allowlist as textfile
`off`	(none)	—	—

Both formats apply the same field allowlist — see lib/exporters/field-allowlist.mjs for the per-schema list. Fields not in the allowlist are dropped before export. This is a CWE-212 mitigation: operator-defined endpoints must never receive accidentally-leaked operator-private data (paths, prompts, brief content).

Environment variables

Variable	Default	Purpose
`VOYAGE_EXPORT_MODE`	`off`	One of `off` / `textfile` / `otlp`
`VOYAGE_TEXTFILE_DIR`	`${CLAUDE_PLUGIN_DATA}`	Directory for `voyage.prom` (textfile mode)
`VOYAGE_OTEL_ENDPOINT`	(none)	HTTPS URL for OTLP/HTTP POST
`VOYAGE_OTEL_ALLOW_PRIVATE`	(unset)	Set to `1` to allow loopback / RFC1918 endpoints
`VOYAGE_TOKEN_METER`	(unset)	Set to a truthy value to capture per-session token/cost into `token-usage-stats.jsonl` on Stop (default off → zero added latency). See Token/cost metering below.

Docker Compose quickstart

A pre-pinned local stack lives at examples/observability/:

cd examples/observability
mkdir -p voyage-textfile
docker compose up -d

This brings up Prometheus, Grafana, node-exporter (textfile mode), and otel-collector (OTLP mode) on localhost. See examples/observability/README.md for endpoint URLs and version pins.

Stats schema

Each Voyage command emits one JSONL record per significant event. Schemas are documented in tests/fixtures/jsonl-schemas.md (Step 1 of v4.1) and locked by tests/lib/profile-stats-fields.test.mjs.

The exporter applies the field allowlist defined in lib/exporters/field-allowlist.mjs. Adding a new field to the JSONL schema does not automatically expose it in OTel — you must add it to the allowlist explicitly. This is intentional: ${CLAUDE_PLUGIN_DATA} is trusted local storage; OTel endpoints are operator-controlled and may be external.

Token/cost metering

SKAL-2. Opt-in capture of per-session token usage and a cache-aware USD cost estimate, folded into the existing Stop hook (hooks/scripts/otel-export.mjs). No new hook is added — capture rides the same Stop event, so there is no second-Stop-hook ordering race.

Activation. Set VOYAGE_TOKEN_METER to any truthy value. When unset (the default) the capture path is skipped entirely — zero added Stop latency. When set, the hook reads the Claude Code transcript (transcript_path from the Stop payload), sums token usage, derives cost, and upserts one record per session into ${CLAUDE_PLUGIN_DATA}/token-usage-stats.jsonl. Capture is fail-open: any error (malformed payload, unreadable transcript) is swallowed and never blocks Stop. Once captured, the record is exported like any other stats file when VOYAGE_EXPORT_MODE is textfile or otlp.

Schema (token-usage). Flat numeric record: ts, session_id, scope, model, tokens_input, tokens_output, tokens_cache_creation, tokens_cache_read, cost_usd, is_estimate, price_table_version. The field allowlist (lib/exporters/field-allowlist.mjs, TOKEN_USAGE_ALLOWED) admits only the numeric + low-cardinality-label fields and strips session_id at export (CWE-212). The exporter auto-promotes each numeric field to a metric: voyage_token_usage_tokens_input (Prometheus) / voyage.token-usage.tokens_input (OTLP).

Cost contract (honesty). cost_usd is computed from a dated, in-source PRICE_TABLE (per-MTok USD), cache-aware: input + output + cache_creation×(5m write rate) + cache_read×(read rate). Each record carries price_table_version (the date the prices were resolved) and is_estimate. When the transcript's model is not in the price table, the meter refuses to guess: cost_usd is null and is_estimate is true. Prices are volatile — re-resolve them against the claude-api reference and bump PRICE_TABLE_VERSION when they change.

v1 limitation — MAIN-CONTEXT only. The meter reads the main-session transcript, which contains only isSidechain:false records. Sub-agent (swarm) turns live in separate agent-*.jsonl sibling files and are not counted, so cost_usd is a lower bound on total Voyage cost, not the session total. Every record is stamped scope:'main-context' to make this explicit. Per-subagent attribution is a documented v2 follow-on (it was a Non-Goal for v1).

Security

The exporter is hardened against three CWE classes:

CWE-22 (path traversal) — lib/exporters/path-validator.mjs rejects relative paths, symlinks, and paths outside allowedRoots (VOYAGE_TEXTFILE_DIR and CLAUDE_PLUGIN_DATA). Tested in tests/hooks/otel-export-validators.test.mjs.
CWE-918 (SSRF) — lib/exporters/endpoint-validator.mjs requires HTTPS, blocks loopback (127.0.0.0/8) and RFC1918 (10/8, 172.16/12, 192.168/16), unless VOYAGE_OTEL_ALLOW_PRIVATE=1 is set explicitly. Cloud metadata endpoints (169.254.169.254) are permanently blocked.
CWE-212 (improper data sanitization) — every record passes through lib/exporters/field-allowlist.mjs before any I/O. Adding a field to the JSONL stream does not expose it externally; operators must update the allowlist intentionally.

Minimum versions per CVE history

Component	Minimum version	Reason
`otel/opentelemetry-collector-contrib`	`0.115.0`	post-CVE-2024-42368
`prom/prometheus`	`3.0.1`	OOM regression fix in 2.x
`prom/node-exporter`	`1.10.2`	textfile collector path normalization
`grafana/grafana`	`11.4.0`	datasource provisioning hardening

Why direct export rather than a native collector

A balance review (docs/voyage-vs-cc-balance-analysis.md §4, V32) asked whether the custom exporters should be dropped in favour of pointing the standard OTEL_* environment variables at a co-located OTLP collector, letting that collector own egress and field selection. The operator decision (D2, 2026-06-20) is to keep direct export. The rationale is the security boundary, not a preference for re-hosting a collector:

The three guards run in-process, before any byte leaves Voyage. path-validator.mjs (CWE-22), endpoint-validator.mjs (CWE-918 / SSRF), and field-allowlist.mjs (CWE-212) are applied inside otel-export.mjs and covered by tests/hooks/otel-export-validators.test.mjs. The records carry operator-private data (paths, prompts, brief content); the allowlist drops everything not explicitly named before export.
A native-collector design moves that boundary out of audited code. Handing raw JSONL to a sidecar collector means either re-expressing the field allowlist in collector config (a second source of truth that can drift) or shipping un-allowlisted private fields and trusting the collector's egress rules. The S21 SSRF hardening — 169.254.169.254 permanently blocked, loopback/RFC1918 gated behind VOYAGE_OTEL_ALLOW_PRIVATE — is a property of endpoint-validator.mjs and would have to be re-created in collector configuration to be preserved.
The collector path is still available, by design. Operators who want collector semantics (retry, persistence, relabelling) use textfile mode and scrape voyage.prom with node-exporter / vector / otel-collector. Direct export is the minimal default, not a rejection of collectors — it keeps the data-sanitization boundary in Voyage's own validated code for the common case.

This is a deliberate direct-export-over-collector choice; the custom exporters and their guards are kept, not pruned.

Limitations

Stop-hook is normal-exit only. If Claude Code crashes or is killed with SIGKILL, the final session's metrics are not flushed. Use --resume on next start to recover plan/progress state; the missing session will not appear in dashboards.
Tail-latency NFR is best-effort. Textfile mode targets <5 ms p99, OTLP <1500 ms (AbortController guards). If the network endpoint is slow, the timeout fires and stats for that session are dropped — the hook always exits 0 to avoid blocking session shutdown.
No retry on transport failure. Stop-hook runs at most once per session. If the OTLP endpoint is unreachable, that session's metrics are lost. Production deployments should use textfile + a robust scrape pipeline (node-exporter, vector, otel-collector with persistent queue) to handle delivery semantics.
No per-tenant labelling. v4.1 emits flat metrics with command and schema_id labels only. Multi-tenant deployments needing per-user or per-project segmentation should layer a relabel stage in their collector or use external metadata.

Cost-estimering disclaimer

The ROUGE-L / Jaccard / character n-gram thresholds in v4.1 are starting points, not contractual SLAs. The brief Risk-tabell explicitly flags these as anslag — they were calibrated against synthetic plan runs (Step 17) using economy and premium profiles. Real cross-tier agreement varies by task complexity. Treat the thresholds as smoke-test floors; tighten them in v4.2 once you have ≥10 production runs of data.

11 KiB Raw Blame History Unescape Escape