open/voyage
1
0
Fork 0
Code Issues Releases 13 Activity

chore(cleanup): R-b R1 — Voyage's own stats, events and OTel export removed

Browse source 
Removes Voyage's measurement, statistics and export apparatus (decision 189;
condition 175 met in step 1) and the R-a leftovers: lib/stats/, lib/exporters/,
the otel-export and post-bash-stats hooks, yardstick, end-state and comparison
gates, synthesis-measure, gold-scorer, docs/observability.md,
examples/observability/, docs/T1-cc26-delegated-orchestration.md and their
tests and fixtures. The six command stats phases and the /trekexecute
main-merge event blocks go; intent-approval --stamp no longer emits
brief-approved. README and docs point to Claude Code's own OpenTelemetry.

Files removed: 52. Files edited: 37.
npm test: 2930 tests (2928 pass / 0 fail / 2 skipped) before, 2600 tests
(2598 pass / 0 fail / 2 skipped) after. removed-surfaces group R1 green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-11-07 20:11:00 +01:00
commit 21480e8627
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
89 changed files with 94 additions and 10290 deletions
Download patch file Download diff file Expand all files Collapse all files

4
CLAUDE.md
View file

@ -30,7 +30,7 @@ Voyage — a contract-driven Claude Code pipeline: brief, research, plan, execut
| Command | Description | Orchestrator model |
|---------|-------------|--------------------|
| `/trekbrief` | Brief — one question round sized by class (`ceremony: small\|bounded\|architectural`, heaviest in doubt, said aloud, only moved up), run in the form of `voyage:interviewing` (asked for by name; criteria by `voyage:writing-criteria`, which `brief-reviewer` is also told to judge by, with `voyage:reviewing-adversarially`), produces a task brief with explicit research plan; ends with the operator explicitly approving `## Intent` + `## Goal` ("sounds good" is not approval), logged in the target repo's hash-chained approval log and committed (the brief too when `artifact_store: git`); the operator's notes from the annotation page go through `voyage:annotating` (`--approve <dir>` does only the approval, then writes the trekbrief record); optionally orchestrates the pipeline | session |
| `/trekbrief` | Brief — one question round sized by class (`ceremony: small\|bounded\|architectural`, heaviest in doubt, said aloud, only moved up), run in the form of `voyage:interviewing` (asked for by name; criteria by `voyage:writing-criteria`, which `brief-reviewer` is also told to judge by, with `voyage:reviewing-adversarially`), produces a task brief with explicit research plan; ends with the operator explicitly approving `## Intent` + `## Goal` ("sounds good" is not approval), logged in the target repo's hash-chained approval log and committed (the brief too when `artifact_store: git`); the operator's notes from the annotation page go through `voyage:annotating` (`--approve <dir>` does only the approval); optionally orchestrates the pipeline | session |
| `/trekresearch` | Research — deep local + external research, produces structured research brief. Opt-in `--engine {swarm\|deep-research}` delegates the external phase to Claude Code's built-in `/deep-research` workflow (swarm default). The external agents preload the `researching-sources` skill and are told it by name at spawn | session |
| `/trekplan` | Plan — brief-reviewer, explore, plan, review. Requires `--brief` or `--project`; halts unless the brief's intent is approved (hash-bound, stale on edit). Auto-discovers `architecture/overview.md` if present. Plan 1.9: the file first on every line — every `Files:` entry and `Test first:` `File:` line opens with its backticked path(s). From 1.8: `Test first:` per step + one `End-to-end:` step; every check written by `voyage:writing-criteria` (named where the plan is written and where its three reviewers start); a design part for `ceremony: architectural` (absent = architectural); critic loop of max 3 rounds, recorded in the plan, rounds 2+ in diff mode (rule A, counted by the gate; rule B in the reviewers), and `plan-gate.mjs` (0 go / 1 stop / 2 could not run) decides; open minors after it are listed for the operator to pick | session |
| `/trekexecute` | Execute — disciplined plan/session-spec executor: refuses a plan whose logged approval does not cover its current sha256 (K2); one fresh `step-implementer` helper per plan step, one controller that reads the diff (spec + quality verdicts), proof of red / floor guard / step check (K9) / `Regression:` (K10) / done line as commands, fix rounds that go up (max 5: helper → plan → operator), architecture rules (K5) + plan-file sweep (K6) after the last step, the app seen with Claude Code's built-in `run` while the `End-to-end:` command stays the verdict (`run` gives no exit code), then its own review (`/simplify` → `/trekreview` → verdict gate, K7; BLOCK keeps the merge into main closed), closed stop list + decision log in the final report | session |
@ -146,7 +146,7 @@ Full flag reference for each command (modes, `--gates`, `--profile`, breaking ch
- **Quality infrastructure (`lib/` validators, parsers, autonomy primitives, hooks):** `docs/architecture.md` §Quality infrastructure
- **Autonomy gates (`--gates`), Path A/B/C decision:** `docs/operations.md`
- **Profile system (`--profile economy/balanced/premium/fable`), lookup order, custom profiles:** `docs/operations.md`
- **Observability (Stop hook, OTLP/textfile export, SSRF mitigation):** `docs/operations.md`
- **Observability:** `docs/operations.md`
- **Handover contracts (the 7 pipeline handovers):** `docs/HANDOVER-CONTRACTS.md`
- **Release and rollback (P13):** `docs/release-runbook.md` — the drilled runbook (before / release / rollback).
`scripts/release-check.mjs --review <review.md>` decides before a release: 0 ready, 1 not ready, 2 never green.

28
README.md
View file

@ -74,7 +74,7 @@ No cloud dependency. No GitHub requirement. Works on **Mac, Linux, and Windows**
All four pipeline commands accept `--gates [true|false|closed|open]`. A bare `--gates`, `true` or `closed` turns gating on; `false`, `open` or no flag (the default) leaves it off. Any other next token is not the flag's value, so `/trekplan --gates --project <dir>` is a bare `--gates`. With gating on, `/trekbrief`, `/trekresearch` and `/trekplan` pause at their own boundaries; `/trekexecute` accepts the flag and pauses on nothing, because its one stop is the merge into main, which always asks. Details: [`docs/operations.md`](docs/operations.md) § Autonomy mode.
Under the hood, `lib/util/autonomy-gate.mjs` maps the `--gates` value to on/off for the four commands, and `lib/stats/event-emit.mjs` records named lifecycle events (the main-merge gate and its answer, among others) to `trekexecute-stats.jsonl` in the plugin data directory. The new `hooks/scripts/post-compact-flush.mjs` PostCompact hook re-injects `.session-state.local.json` after context compaction so multi-session work survives a compaction boundary.
Under the hood, `lib/util/autonomy-gate.mjs` maps the `--gates` value to on/off for the four commands. The new `hooks/scripts/post-compact-flush.mjs` PostCompact hook re-injects `.session-state.local.json` after context compaction so multi-session work survives a compaction boundary.
## Quick start
@ -173,7 +173,7 @@ Borrowed, with credit: size classes and the ratchet from obra/superpowers (MIT),
| **Default** | `/trekbrief <task>` | Question round sized by class, then review and approval. |
| **Quick** | `/trekbrief --quick <task>` | Proposes `small` as the class; still said aloud, overridable, and raised if the task proves bigger. |
| **Profile** | `/trekbrief --profile <name> <task>` | (v4.1.0) Pin model profile for the brief phase: `economy` / `balanced` / `premium` / `fable` / `<custom>`. See [Profile system](#profile-system-v410) below. |
| **Approve** | `/trekbrief --approve <project-dir \| brief-file>` | No interview: show an existing brief's `## Intent` + `## Goal` verbatim and ask the operator to approve them; on "Approve" it also writes the trekbrief stats record. The same step (Phase 4h) ends every normal `/trekbrief` run. |
| **Approve** | `/trekbrief --approve <project-dir \| brief-file>` | No interview: show an existing brief's `## Intent` + `## Goal` verbatim and ask the operator to approve them. The same step (Phase 4h) ends every normal `/trekbrief` run. |
`/trekbrief` is **always interactive**. There is no foreground/background mode — the interview requires user input.
@ -482,14 +482,6 @@ session-picker output trivially identifiable. Slug derivation:
The hook is fail-open — any error → title is left untouched.
### Per-step timing (CC v2.1.97+)
A `PostToolUse` hook (`hooks/scripts/post-bash-stats.mjs`) appends
`duration_ms` from each Bash tool call to
`${CLAUDE_PLUGIN_DATA}/trekexecute-stats.jsonl`. One line per Bash
call; useful for identifying long-running verify or checkpoint commands
across executions.
---
## `/trekreview` — Review
@ -964,21 +956,13 @@ A brief's `phase_signals` (written by `/trekbrief`) can set the model for resear
See [`docs/profiles.md`](docs/profiles.md) for the decision tree, custom-profile authoring, and cost estimation disclaimer (the per-profile cost numbers are estimates, not contractual SLAs).
## Observability (v4.1.0)
## Observability
Stop-hook OTel/Prometheus export — opt-in via `VOYAGE_EXPORT_MODE`:
| Mode | Output | Endpoint env-var |
|------|--------|------------------|
| `off` (default) | _(no export)_ | — |
| `textfile` | `voyage.prom` (Prometheus exposition format) | `VOYAGE_TEXTFILE_DIR` |
| `otlp` | OTLP/JSON POST | `VOYAGE_OTEL_ENDPOINT` |
Default JSONL stats stream (`${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl`) is unchanged — OTel is additive. Local Docker Compose stack: [`examples/observability/`](examples/observability/). Operator docs: [`docs/observability.md`](docs/observability.md). Both pin minimum versions per CVE history.
Claude Code's own OpenTelemetry (`CLAUDE_CODE_ENABLE_TELEMETRY`) gives tool durations, tokens and cost per agent and skill: see [Monitoring usage](https://code.claude.com/docs/en/monitoring-usage) (read 2026-11-07, Claude Code 2.1.319).
## Cost profile
The default `premium` profile runs **Opus on every phase** of the pipeline's agent work — the exploration and review swarms (5–10 sub-agents per command; spawn sites inject the composed brief > profile > frontmatter resolution, with `agents/*.md` `model: opus` pins as the fallback) and the executor (one per plan session). The command orchestrator itself is not profile-controlled: as of v5.9, command frontmatter omits `model:`, so the orchestrator follows the session model. For cheaper runs, opt into `--profile balanced` (Sonnet on brief/research/execute/continue, Opus on plan + review) or `--profile economy` (Sonnet everywhere); for maximum quality, `--profile fable` (Fable 5.1 on every phase). Per-command cost is published in `${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl` if you want exact numbers.
The default `premium` profile runs **Opus on every phase** of the pipeline's agent work — the exploration and review swarms (5–10 sub-agents per command; spawn sites inject the composed brief > profile > frontmatter resolution, with `agents/*.md` `model: opus` pins as the fallback) and the executor (one per plan session). The command orchestrator itself is not profile-controlled: as of v5.9, command frontmatter omits `model:`, so the orchestrator follows the session model. For cheaper runs, opt into `--profile balanced` (Sonnet on brief/research/execute/continue, Opus on plan + review) or `--profile economy` (Sonnet everywhere); for maximum quality, `--profile fable` (Fable 5.1 on every phase).
On the Anthropic API the `opus` alias resolves to **Opus 5.5** and `sonnet` to **Sonnet 5.5** (both default to reasoning effort `medium`), and `fable` to **Fable 5.1** (the most capable model in Claude Code; default reasoning effort `high` — xhigh must be chosen explicitly, see [`docs/profiles.md`](docs/profiles.md)). Other providers resolve `opus` and `sonnet` to other versions. Note two distinct effort axes that share the word "effort": brief `phase_signals.effort` (low/standard/high) tunes *orchestration shape* — how many agents and passes run — while native `effort:` (nine agents pin `high`; the rest follow the session) tunes the *per-spawn reasoning budget*. See [`docs/profiles.md`](docs/profiles.md) § Model & effort axes.
@ -1002,7 +986,7 @@ voyage/
│ └ reviewing-adversarially: the review agents; writing-criteria: brief-reviewer, plan-critic; interviewing: named by /trekbrief;
│ └ annotating: named by /trekbrief, /trekplan and /trekreview)
├── templates/ Frontmatter templates for brief, research, plan, session
├── hooks/ 12 hooks (pre-bash, pre-write, pre-agent-cap, pre-plan-gate, session-title, post-bash-stats, post-approval-log, post-edit-check, pre-compact-flush, post-compact-flush, otel-export, stop-test-lock)
├── hooks/ 10 hooks (pre-bash, pre-write, pre-agent-cap, pre-plan-gate, session-title, post-approval-log, post-edit-check, pre-compact-flush, post-compact-flush, stop-test-lock)
├── lib/ Zero-dep parsers and validators (CLI shims under lib/validators/)
├── tests/ comprehensive node:test suite — `npm test` is the fork-readiness gate
├── docs/ HANDOVER-CONTRACTS.md, architecture.md, operations.md, profiles.md, command-modes.md, release-runbook.md

78
commands/trekbrief.md
View file

@ -72,8 +72,7 @@ Parse `$ARGUMENTS`:
On exit 1, print its `error` and stop. Otherwise set `BRIEF_PATH`,
`PROJECT_DIR` and `PLAN_COMMAND` from `brief_path`, `project_dir` and
`plan_command`, jump straight to **Phase 4h**, run it and print its
report; if the stamp printed `stamped: true`, run **Phase 7** with the
approve-mode values it names, then stop. This is the remedy `/trekplan` names when it halts on an
report, then stop. This is the remedy `/trekplan` names when it halts on an
unapproved or stale intent, on both of its brief modes.
In every other mode, `BRIEF_PATH = {PROJECT_DIR}/brief.md` and
@ -923,8 +922,7 @@ node ${CLAUDE_PLUGIN_ROOT}/lib/validators/intent-approval.mjs --stamp "{BRIEF_PA
```
This writes `intent_approved_hash` (sha256 over the normalized `## Intent`
+ `## Goal`) and `intent_approved_at` into the frontmatter, and emits the
`brief-approved` lifecycle event. It prints JSON; if `stamped` is `false`
+ `## Goal`) and `intent_approved_at` into the frontmatter. It prints JSON; if `stamped` is `false`
(e.g. `## Goal` is empty), report the `reason` and return to step 1. Editing
`## Intent` or `## Goal` afterwards makes the approval stale — `/trekplan`
halts until the operator approves the new text.
@ -969,16 +967,14 @@ Logged: {record {seq} in .claude/voyage/approvals.jsonl | no — {reason}}
Committed: {{short sha}: brief + log | {short sha}: log only, brief kept on this machine | no — {reason}}
```
In `--approve` mode, run **Phase 7** (`"mode": "approve"`) when the stamp
printed `stamped: true`, then stop and print the next step:
In `--approve` mode, stop and print the next step:
`{PLAN_COMMAND}` (only if approved).
## Phase 5 — Auto-orchestration opt-in (if research_topics > 0)
**With research_topics = 0**, print the manual-path output below without the
`/trekresearch` lines (Brief, then `/trekplan --project {PROJECT_DIR}`, then
`/trekexecute --project {PROJECT_DIR}`), then run Phase 7 (`auto_result: manual`)
and stop. Phase 6 is never entered without the operator's opt-in.
`/trekexecute --project {PROJECT_DIR}`), then stop. Phase 6 is never entered without the operator's opt-in.
Ask the user via `AskUserQuestion`:
@ -1012,23 +1008,19 @@ Then:
/trekexecute --project {PROJECT_DIR}
```
Run Phase 7 (`auto_result: manual`), then stop. Do not continue to Phase 6.
Stop. Do not continue to Phase 6.
### Auto path
**Auto requires an approved intent.** If Phase 4h ended "Leave unapproved",
the auto chain would halt at `/trekplan` anyway — print the manual-path
output instead, run Phase 7 (`auto_result: manual`) and stop.
output instead and stop.
Set `auto_research: true` in the brief's frontmatter (edit the file).
The `brief-approved` lifecycle event was already emitted by the Phase 4h
stamp — on both paths — so it is not emitted again here.
If `gates_mode == true`: pause here via `AskUserQuestion` —
"Auto-mode confirmed. Proceed to research now? (yes/no)". If the user
answers no, fall back to the manual path output, run Phase 7
(`auto_result: cancelled`) and stop. Otherwise proceed to Phase 6.
answers no, fall back to the manual path output and stop. Otherwise proceed to Phase 6.
If `gates_mode == false` (default in auto): proceed directly to Phase 6.
The chain ends when `/trekplan` has written the plan (Step 6e); it never
@ -1052,10 +1044,9 @@ Continue?"
| Option | Action |
|--------|--------|
| **Continue — auto** | Proceed. |
| **Cancel — do manual** | Revert to manual path (run Phase 7 with `auto_result: cancelled`, print commands, stop). |
| **Cancel — do manual** | Revert to manual path (print commands, stop). |
If cancelled → fall back to manual path output, run Phase 7
(`auto_result: cancelled`) and stop.
If cancelled → fall back to manual path output and stop.
### Step 6b — Run research topics sequentially (inline)
@ -1091,8 +1082,7 @@ ls -1 {PROJECT_DIR}/research/*.md | wc -l
```
Expected count: N. If any are missing, report and ask the user how to
proceed (retry, skip missing topic, cancel). On cancel, run Phase 7
(`auto_result: cancelled`) and stop.
proceed (retry, skip missing topic, cancel). On cancel, stop.
Update brief frontmatter: `research_status: complete`.
@ -1107,7 +1097,7 @@ Invoke the planning command inline in this session:
The planning pipeline runs all phases (exploration, synthesis, review) in
main context. Wait for the plan to be written to `{PROJECT_DIR}/plan.md`
before continuing. If `/trekplan` stops before it writes the plan, report
where it stopped, run Phase 7 (`auto_result: failed`) and stop.
where it stopped and stop.
### Step 6e — Report completion
@ -1137,52 +1127,6 @@ Or:
/trekexecute --validate --project {PROJECT_DIR} # schema check
```
## Phase 7 — Stats tracking
Append one record to `trekbrief-stats.jsonl` in the plugin data directory.
Run it exactly like this. The record goes on stdin, and the writer resolves
the directory: `CLAUDE_PLUGIN_DATA`, which the Bash tool env does not carry,
else the directory `scripts/yardstick.mjs` reads.
```bash
node ${CLAUDE_PLUGIN_ROOT}/lib/stats/stats-append.mjs trekbrief <<'JSON'
{
"ts": "{ISO-8601}",
"task": "{task description (first 100 chars)}",
"slug": "{slug}",
"mode": "{default | quick | approve}",
"interview_turns": {N},
"review_iterations": {1..3},
"brief_quality": "{complete | partial}",
"research_topics": {N},
"auto_research": {true | false},
"auto_result": "{completed | cancelled | failed | manual}",
"intent_approved": {true | false},
"project_dir": "{path}"
}
JSON
```
`intent_approved` is `true` only when Phase 4h ran the stamp (the operator
answered "Approve"). `slug` is the brief's `slug`, the same one the stamp's
`brief-approved` record and `/trekplan`'s record carry.
In `--approve` mode the record carries fixed values: `"mode": "approve"`,
`"interview_turns": 0`, `"review_iterations": 0`, `brief_quality`,
`research_topics` and `auto_research` from the brief's frontmatter (`complete`,
`0`, `false` when absent), `"auto_result": "manual"`, `"intent_approved": true`.
The writer adds three keys itself: `order_id`, `order_claimed_at` and
`order_reason` - the coord-queue order this run is working on, read from the
queue's claimed orders (exactly one: its id and claim time; zero or several:
`order_id: null` and the reason). Never type them: any `order_*` key in the
record is replaced. `scripts/yardstick.mjs` joins its `use` number on
`order_id`.
The writer prints `{"written": true, ...}`. On `"written": false` (exit 1),
print its `reason` in one line and continue. A stats failure never blocks the
workflow, but it is reported, not skipped silently.
## Profile (v4.1)
Accepts `--profile <name>` where `<name>` is one of `economy`, `balanced`,

17
commands/trekcontinue.md
View file

@ -268,23 +268,6 @@ If the brief file does not exist (validator emits a warning but does not
fail), print: `Warning: next_session_brief_path "{path}" does not exist on
disk. Cannot continue automatically.` and exit. Do not guess.
## Phase 5 — Stats tracking
Append one record to `trekcontinue-stats.jsonl` in the plugin data directory.
Run it exactly like this. The record goes on stdin, and the writer resolves
the directory: `CLAUDE_PLUGIN_DATA`, which the Bash tool env does not carry,
else the directory `scripts/yardstick.mjs` reads.
```bash
node ${CLAUDE_PLUGIN_ROOT}/lib/stats/stats-append.mjs trekcontinue <<'JSON'
{"ts":"<iso-8601>","project":"<project>","next_session_label":"<label>","status":"<status>"}
JSON
```
The writer prints `{"written": true, ...}`. On `"written": false` (exit 1),
print its `reason` in one line and continue. A tracking failure never blocks
the main workflow, but it is reported, not skipped silently.
## Profile (v4.1) — inheritance from plan-frontmatter
Accepts `--profile <name>` where `<name>` is `economy`, `balanced`, `premium`,

97
commands/trekexecute.md
View file

@ -2575,29 +2575,6 @@ Always produce a final report.
### Main-merge gate (MAIN_MERGE_GATE — always pauses)
Before writing the final progress + state files (and before any "merge to
main" prose), emit the `main-merge-gate` lifecycle event so observability
and operator tooling can see that the pipeline reached its terminal
boundary:
```bash
node ${CLAUDE_PLUGIN_ROOT}/lib/stats/event-emit.mjs --event main-merge-gate --payload - <<'JSON'
{"plan":"{plan-path}","wave_count":{wave-count}}
JSON
```
Fill in the placeholders yourself. The payload goes on stdin through a quoted
heredoc, so the shell expands nothing in it: not a variable (shell state from an
earlier Bash call does not survive, and an expanded variable is how this event
used to print `payload-not-json`), and not a quote character or `$(...)` in the
plan path. `{plan-path}` is the repo-relative plan path written as a JSON
string (escape a `"` or `\` in it). `{wave-count}` is the number of merged
waves, `1` on the single-session path.
Run each event call as its own command, never chained with `&&`. The writer
prints `{"written": true, ...}`. On `"written": false` (exit 1), print its
`reason` in one line and continue — an event failure never blocks the gate.
**The gate is a lock, not a question** (operator decision 2026-10-08). The
guard (`pre-bash-executor`) asks the operator before every merge into main and
every push, in auto mode too. When the executor first asked "ready to merge?"
@ -2606,8 +2583,7 @@ merge may run at all, and the guard's question is the one question.
**Only the closing run passes the lock** (Phase 7.2: a single-session run of
the whole plan, or the wave parent after the last wave). A `--session N`
child, a `--step N` run and a session spec deliver a part: they emit no gate
event, merge nothing, and their result is left as Phase 7.5 decided it
child, a `--step N` run and a session spec deliver a part: they merge nothing, and their result is left as Phase 7.5 decided it
(`main_merge: "n/a"`). A wave session's `completed` must reach its parent as
`completed`, or Step 3 would read the wave as failed.
@ -2641,9 +2617,9 @@ has confirmed. Any other exit 2 that does not start `review-stale` or
is reported as "not measured", never green, and the merge stays locked with
the decline reason `review-not-measured`.
The merge runs only on exit 0 and a `completed` result. **Closed:** emit
`main-merge-declined` with the reason `review-block`, `review-stale`,
`review-unanswered`, `review-not-measured` or `run-not-completed`. Nothing is finalized: a `completed` result becomes
The merge runs only on exit 0 and a `completed` result. **Closed** (reason
`review-block`, `review-stale`,
`review-unanswered`, `review-not-measured` or `run-not-completed`). Nothing is finalized: a `completed` result becomes
`partial` (`failed` and `stopped` stay), and the commits stay where they are
for review. Say in one line why it is closed and what opens it: the
BLOCKERs fixed through Phase 7.2's loop, a review that runs in full, a review
@ -2660,9 +2636,9 @@ git worktree list --porcelain
```
- **No `MAIN=` line** (the repo has neither branch): there is nothing to merge
into. Emit `main-merge-declined` with the reason `no-main-branch`.
into (reason `no-main-branch`).
- **The run's branch IS main (or master):** there is nothing to merge; the
commits are already on main. Emit `main-merge-approved` and say in the
commits are already on main. Say in the
report that the review ran after the commits landed on main and could not
hold them back: run on a branch when the review should gate the merge.
- **Another branch:** first, before the merge call, write the run evidence
@ -2713,20 +2689,19 @@ git worktree list --porcelain
as a `merge` approval (K2). The executor asks nothing before or after it. In
the permission mode where Claude Code asks nothing, the guard refuses the
call instead, because there is no one to ask (operator decision 2026-10-09).
- `MERGED` → emit `main-merge-approved`.
- `MERGED` → the merge ran.
- The call did not run (the operator declined at the guard's question, or
the guard refused it) → emit `main-merge-declined` with the reason
the guard refused it) → reason
`hold-for-review`; the run is `partial`, and the commits stay on the run's
branch. When the guard refused, say so, and that the operator merges by
hand.
- `MERGE_CONFLICT` → list the conflicting files the block printed (the
merge is already aborted), emit `main-merge-declined` with the reason
merge is already aborted), reason
`merge-conflict`; the run is `partial`. Never resolve a conflict with
`-X ours/theirs`.
- `MERGE_REFUSED` → git refused to start (local changes in the main tree, or
untracked files in the way: its message says which). Emit
`main-merge-declined` with the reason `merge-refused` and report git's
message; the run is `partial`.
untracked files in the way: its message says which). Report git's
message (reason `merge-refused`); the run is `partial`.
This gate is **always on**, regardless of `gates_mode`: a lock that every run
passes through, the one boundary that ALWAYS pauses on every run, either at
@ -2739,11 +2714,6 @@ and the merged-waves record is kept until the confirm below.
On confirm:
(the merge ran, or there was nothing to merge)
```bash
node ${CLAUDE_PLUGIN_ROOT}/lib/stats/event-emit.mjs --event main-merge-approved --payload - <<'JSON'
{"plan":"{plan-path}"}
JSON
```
On the multi-session wave path (Phase 2.6), and only when `next-wave` said
`none` (every wave merged) and Step 4 passed, also clear the plan's orders
@ -2756,17 +2726,8 @@ cleared:
On decline:
(a closed lock, or the guard's question declined, or a conflict)
```bash
node ${CLAUDE_PLUGIN_ROOT}/lib/stats/event-emit.mjs --event main-merge-declined --payload - <<'JSON'
{"plan":"{plan-path}","reason":"{reason}"}
JSON
```
The reason is one of the literal words above (`review-block`,
`review-stale`, `review-unanswered`, `review-not-measured`, `run-not-completed`, `no-main-branch`, `hold-for-review`,
`merge-conflict`, `merge-refused`), never the operator's free text (free text breaks the JSON).
On decline, nothing is cleared: the merged-waves record is kept.
Nothing is cleared: the merged-waves record is kept.
**Recovery surface:** on the single-session path, if declined,
`--resume re-enters at the gate` — re-running
@ -3013,36 +2974,6 @@ run resumes at the lowest wave not yet merged, or re-runs Step 4 and returns to
the gate, and the `clear` command in Step 0 starts the plan over from its first
wave instead.
## Phase 9 — Stats tracking
Append one record to `trekexecute-stats.jsonl` in the plugin data directory.
Run it exactly like this. The record goes on stdin, and the writer resolves
the directory: `CLAUDE_PLUGIN_DATA`, which the Bash tool env does not carry,
else the directory `scripts/yardstick.mjs` reads.
```bash
node ${CLAUDE_PLUGIN_ROOT}/lib/stats/stats-append.mjs trekexecute <<'JSON'
{
"ts": "{ISO-8601}",
"plan": "{filename only}",
"plan_type": "{plan | session-spec}",
"mode": "{execute | resume | dry-run | step}",
"result": "{completed | failed | stopped | partial}",
"steps_total": 0,
"steps_passed": 0,
"steps_failed": 0,
"steps_skipped": 0,
"failed_at_step": null
}
JSON
```
`steps_passed` counts the steps whose status is `completed`.
The writer prints `{"written": true, ...}`. On `"written": false` (exit 1),
print its `reason` in one line and continue. A stats failure never blocks the
workflow, but it is reported, not skipped silently.
## Profile (v4.1)
Accepts `--profile <name>` where `<name>` is `economy`, `balanced`, `premium`,
@ -3094,7 +3025,7 @@ model — that is fixed at invocation time (command frontmatter omits
`/trekexecute` spawns no swarm: one step helper at a time and one controller
(Hard Rule 10), and both run `opus` — the step agent's pin and the
controller's `model: "opus"` — so `phase_signal_result.model` still has no
spawn site here; it is returned for cross-command uniformity; no stats record carries it.
spawn site here; it is returned for cross-command uniformity.
For `/trekexecute` specifically: `effort == 'low'` runs the plan as one
session, step after step (Phase 2.5). `effort == 'standard'` (or absent) →
@ -3196,7 +3127,7 @@ set from the effort.
15. **No writing outside the repository.** During step execution, never write
files outside the git repository root (`git rev-parse --show-toplevel`).
Exception: `.claude/` paths for plans, progress files, and stats,
Exception: `.claude/` paths for plans and progress files,
AND `~/.claude/projects/*/memory/` paths for memory files when the
step's manifest declares `memory_write: true`. The `memory_write: true`
opt-in is a second gate: unauthorized memory writes still fail. This

42
commands/trekplan.md
View file

@ -1293,51 +1293,13 @@ Confirm the plan and brief file locations and exit.
Do not implement in this session. Print the line the operator gives
`/trekexecute` — `/trekexecute --project {project_dir}`, or
`/trekexecute {plan_path}` without a project directory — then run Phase 12 (`outcome: execute`).
`/trekexecute {plan_path}` without a project directory.
`/trekexecute` is what checks the plan's approval (K2), locks the tests, runs
one step helper per step and the review.
### "execute with team" / "team"
There is no team path. Print the same `/trekexecute` line, then run Phase 12
(`outcome: execute_team`); parallel work is `--decompose` followed by `/trekexecute`'s session orders.
## Phase 12 — Session tracking
After the plan is presented (Phase 10) or after handoff (Phase 11), append a
session record to `trekplan-stats.jsonl` in the plugin data directory. Run it
exactly like this. The record goes on stdin, and the writer resolves the
directory: `CLAUDE_PLUGIN_DATA`, which the Bash tool env does not carry, else
the directory `scripts/yardstick.mjs` reads.
```bash
node ${CLAUDE_PLUGIN_ROOT}/lib/stats/stats-append.mjs trekplan <<'JSON'
{
"ts": "{ISO-8601 timestamp}",
"task": "{task description (first 100 chars)}",
"mode": "{default|fg|quick}",
"slug": "{the brief's slug}",
"brief_path": "{brief_path}",
"project_dir": "{project_dir or null}",
"codebase_size": "{small|medium|large}",
"codebase_files": {N},
"agents_deployed": {N},
"deep_dives": {N},
"research_briefs_used": {N},
"research_scout_used": {true|false},
"critic_verdict": "{BLOCK|REVISE|PASS}",
"guardian_verdict": "{ALIGNED|CREEP|GAP|MIXED}",
"outcome": "{execute|execute_team|save|refine}"
}
JSON
```
`slug` is the brief's `slug`, so this record pairs with the brief's
`brief-approved` and `trekbrief` records. `critic_verdict` is the `verdict` of
the last round `plan-gate.mjs --record` wrote into the plan. The writer prints
`{"written": true, ...}`. On `"written": false` (exit 1), print its `reason` in
one line and continue. A tracking failure never blocks the main workflow, but
it is reported, not skipped silently.
There is no team path. Print the same `/trekexecute` line; parallel work is `--decompose` followed by `/trekexecute`'s session orders.
## Profile (v4.1)

65
commands/trekresearch.md
View file

@ -292,7 +292,7 @@ Synthesize findings inline. Write a lightweight research brief to the destinatio
path, following the research-brief-template but with shorter sections and fewer
dimensions.
Skip to Phase 8 (stats tracking) after writing the brief.
Skip to Phase 8 after writing the brief.
## Phase 4 — Parallel research (agent swarm)
@ -492,7 +492,7 @@ agent surfaced that no interview dimension claims.
up enforcing different numbers for one ceiling.
4. **Record the baseline.** Keep the interview-derived count as
`dimensions_baseline` so the discovered delta is machine-readable against
the final `dimensions` (Phase 8 stats).
the final `dimensions`.
5. **Attest membership, not just the count.** Set
`dimensions_baseline_preserved: true` only if EVERY interview-derived
dimension is still on the final list — this phase appends, it never replaces.
@ -791,7 +791,7 @@ one dimension), set `effective_engine = swarm`, fall back to the swarm engine fo
this run (and log the fallback per the Engine selection step), rather than emit an
invalid brief.
## Phase 8 — Present and track
## Phase 8 — Present
Present a summary to the user:
@ -825,60 +825,6 @@ You can:
- Ask follow-up questions about specific findings
```
### Stats tracking
Append one record to `trekresearch-stats.jsonl` in the plugin data directory.
Run it exactly like this. The record goes on stdin, and the writer resolves
the directory: `CLAUDE_PLUGIN_DATA`, which the Bash tool env does not carry,
else the directory `scripts/yardstick.mjs` reads.
```bash
node ${CLAUDE_PLUGIN_ROOT}/lib/stats/stats-append.mjs trekresearch <<'JSON'
{
"ts": "{ISO-8601 timestamp}",
"question": "{research question (first 100 chars)}",
"mode": "{default|quick}",
"scope": "{both|local|external}",
"engine": "{effective engine: swarm|deep-research}",
"slug": "{brief slug}",
"project_dir": "{project_dir or null}",
"brief_path": "{brief_destination}",
"dimensions": {N},
"dimensions_baseline": {N},
"dimensions_baseline_preserved": {true|false},
"effort": "{low|standard|high}",
"conv_turns": {N},
"empty_turns": {N},
"unique_sources": {N},
"agents_local": {N},
"agents_external": {N},
"gemini_used": false,
"confidence": {0.0-1.0},
"contradictions": {N},
"open_questions": {N}
}
JSON
```
**The six measurement fields (v5.10).** `effort` is the grouping key — the
resolved `phase_signal_result.effort` for the `research` phase, a
low-cardinality label (`low|standard|high`), and the only axis on which a
high-effort run can be compared against a standard one. Four are numeric:
`unique_sources` (distinct sources cited across the brief),
`dimensions_baseline` (the interview-derived dimension count, so the Phase 4.5
delta against `dimensions` is machine-readable), `conv_turns` (Phase 5 loop
turns actually spent), and `empty_turns` (loop turns that returned no findings
or no citations). The sixth is boolean: `dimensions_baseline_preserved`, the
Phase 4.5 attestation (step 5) that every interview dimension survived onto the
final list — the count delta cannot show membership, and the dimension NAMES
that could are prose the exporter allowlist denies. On a standard run the loop
never arms, so `dimensions_baseline == dimensions`, both turn counters are `0`,
and `dimensions_baseline_preserved` is `true` (nothing touched the list).
The writer prints `{"written": true, ...}`. On `"written": false` (exit 1),
print its `reason` in one line and continue. A tracking failure never blocks
the main workflow, but it is reported, not skipped silently.
## Profile (v4.1)
Accepts `--profile <name>` where `<name>` is `economy`, `balanced`, `premium`, `fable`,
@ -900,8 +846,6 @@ Examples:
VOYAGE_PROFILE=balanced /trekresearch
```
Stats records emit `engine` so operators can audit which engine drove which session.
## Composition rule (v5.1)
Independent of the profile system. When `brief.md` carries
@ -953,8 +897,7 @@ always carries an adversarial counter-evidence pass.
The former third-party second-opinion bridge pass is **unavailable,
skipped** (dropped in v5.10.1 by operator directive). High effort no
longer has a bridge arm; `gemini_used` is a vestigial stats field
pinned to `false`.
longer has a bridge arm.
High effort additionally arms the Phase 5 bounded follow-up loop (max 3
turns per under-illuminated dimension, budgeted by

23
commands/trekreview.md
View file

@ -190,7 +190,7 @@ trailing block `{"verdict": "ALLOW", "counts": {"BLOCKER": 0, "MAJOR": 0,
without it the verdict gate
cannot back the ALLOW and exits 2. There is nothing for the Phase 4.7 engine
to review, so its absence does not withhold ALLOW here. Skip Phases 4–7.
Continue to Phase 8 (validate + stats).
Continue to Phase 8 (validate).
## Phase 4 — Triage gate (deterministic path-pattern classifier)
@ -679,7 +679,7 @@ to a plan without rewording them; earlier steps are never rewritten.
If `mode == dry-run`: skip the write; print the would-be path and the
first 60 lines of the rendered output.
## Phase 8 — Validate output + stats
## Phase 8 — Validate output
Run the strict validator:
@ -723,22 +723,7 @@ with WARN, `--json` also returns `warn_reasons` (the JSON block's
as returned. This is the command that later stops a merge or a release on
BLOCK; `/trekreview` reports it and never overrides it.
Then append one record to `trekreview-stats.jsonl` in the plugin data directory.
Run it exactly like this. The record goes on stdin, and the writer resolves
the directory: `CLAUDE_PLUGIN_DATA`, which the Bash tool env does not carry,
else the directory `scripts/yardstick.mjs` reads.
```bash
node ${CLAUDE_PLUGIN_ROOT}/lib/stats/stats-append.mjs trekreview <<'JSON'
{"ts":"{ISO-8601}","slug":"{slug}","verdict":"BLOCK|WARN|ALLOW","counts":{"BLOCKER":N,"MAJOR":N,"MINOR":N,"SUGGESTION":N},"reviewed_files_count":N,"mode":"default|quick|validate|dry-run","duration_ms":N}
JSON
```
The writer prints `{"written": true, ...}`. On `"written": false` (exit 1),
print its `reason` in one line and continue. A tracking failure never blocks
the main workflow, but it is reported, not skipped silently.
**Build the operator-annotation HTML.** After stats land, run:
**Build the operator-annotation HTML.** After the gate, run:
```bash
ANNOT_HTML=$(node ${CLAUDE_PLUGIN_ROOT}/scripts/annotate.mjs "{review_path}" 2>&1)
@ -1056,7 +1041,7 @@ Low effort: skip the brief-conformance reviewer entirely (existing
- **Honesty.** If the diff is trivially small or all-skip, say so. Do
not pad findings to make the review look thorough.
- **No production code.** This command never runs production code, never
writes to anything outside `{project_dir}` and `${CLAUDE_PLUGIN_DATA}` —
writes to anything outside `{project_dir}` —
with one exception, only after the operator approved that exact line in
Phase 8.7: Voyage's own marked block in `.claude/rules/voyage-review-lessons.md`
and a one-line pointer in `CLAUDE.md`.

2
docs/HANDOVER-CONTRACTS.md
View file

@ -110,7 +110,7 @@ Optional but standard sections: `## Non-Goals`, `## Constraints`, `## Preference
**Pre-2.2 briefs receive zero framing enforcement (producer-elective defense).** The framing-alignment defense is gated at `brief_version ≥ 2.2`, so a brief declaring `2.0` or `2.1` gets **none** of it: `framing` is optional and a missing value is not an error, the `## TL;DR` section is not required, and the `brief-reviewer` memory-alignment dimension scores N/A. The backward-compatibility this buys is deliberate, but it also means the framing defense is **producer-elective**: any producer can sidestep the entire defense — deliberately or by shipping a legacy template — simply by declaring `brief_version: "2.1"`, and the pipeline raises no signal by default. This is a documented property of the contract, not a bug. Two remedies close it for a given consumer: (1) require upstream producers to emit `brief_version: "2.2"` (which forces `framing` + `## TL;DR`); or (2) pass the opt-in **`--min-brief-version <ver>`** flag to `/trekplan` or `/trekresearch` (forwarded to the validator as `--min-version`), which raises a `BRIEF_VERSION_BELOW_MINIMUM` **warning** — never a block — when a brief declares a version below the floor. The flag defaults to off, so the N-1 compatibility window for `2.0`/`2.1` briefs is preserved unless a consumer explicitly opts into the stricter floor.
**Intent approval gate (veikart steg 1) — additive schema, consumer-side requirement.** The two `intent_approved_*` fields are additive and optional in the schema: `brief-validator.mjs` does not require them, `/trekresearch` does not read them, and no `brief_version` bump is involved. The requirement lives in `/trekplan` Phase 1 (`### Read the brief`), which runs `intent-approval.mjs --check` and **halts** on `BRIEF_INTENT_NOT_APPROVED` (no marker), `BRIEF_INTENT_APPROVAL_STALE` (`## Intent` or `## Goal` changed after approval) or `BRIEF_INTENT_APPROVAL_INVALID` (malformed marker, or a section missing, empty or present more than once; a `## ` line inside a code block or HTML comment is not a heading). It is version-independent, so a brief from ANY producer, of any `brief_version`, stops at `/trekplan` until approved. No producer has to emit anything new: the operator approves through `/trekbrief --approve <project-dir | brief-file>`, a Voyage-side step open to every producer's brief, whether it is planned with `--project` (`<dir>/brief.md`) or `--brief <file>`, so no producer is privileged. `trekreview` briefs are exempt. The exemption is read from the brief's own frontmatter, so relabelling a brief `type: trekreview` walks past the gate, the same trust class as self-stamping. **What the marker does not prove:** same user, same machine. Any session can run `--stamp`. The marker is a trace that the approval step ran against this exact text, not a signature (see the header of `lib/validators/intent-approval.mjs`). `--stamp` also emits `brief-approved` to `trekexecute-stats.jsonl` under `CLAUDE_PLUGIN_DATA`, falling back to the plugin data dir the yardstick reads, on both the manual and the auto path.
**Intent approval gate (veikart steg 1) — additive schema, consumer-side requirement.** The two `intent_approved_*` fields are additive and optional in the schema: `brief-validator.mjs` does not require them, `/trekresearch` does not read them, and no `brief_version` bump is involved. The requirement lives in `/trekplan` Phase 1 (`### Read the brief`), which runs `intent-approval.mjs --check` and **halts** on `BRIEF_INTENT_NOT_APPROVED` (no marker), `BRIEF_INTENT_APPROVAL_STALE` (`## Intent` or `## Goal` changed after approval) or `BRIEF_INTENT_APPROVAL_INVALID` (malformed marker, or a section missing, empty or present more than once; a `## ` line inside a code block or HTML comment is not a heading). It is version-independent, so a brief from ANY producer, of any `brief_version`, stops at `/trekplan` until approved. No producer has to emit anything new: the operator approves through `/trekbrief --approve <project-dir | brief-file>`, a Voyage-side step open to every producer's brief, whether it is planned with `--project` (`<dir>/brief.md`) or `--brief <file>`, so no producer is privileged. `trekreview` briefs are exempt. The exemption is read from the brief's own frontmatter, so relabelling a brief `type: trekreview` walks past the gate, the same trust class as self-stamping. **What the marker does not prove:** same user, same machine. Any session can run `--stamp`. The marker is a trace that the approval step ran against this exact text, not a signature (see the header of `lib/validators/intent-approval.mjs`).
**Approval log (K2) — consumer-side requirement, no schema change.** After the stamp, `/trekbrief` appends the approval to the target repo's log with `lib/util/approval-log.mjs append --kind intent --subject <brief> --sha256 <intent hash>` and commits it with `lib/util/artifact-commit.mjs` (the log always; the brief too when `artifact_store: git`). The commit carries the caller's trailer: every call passes `--trailer "Co-Authored-By: Claude {model} <noreply@anthropic.com>"` with `{model}` filled in, and the CLI refuses (exit 1, nothing committed) a missing trailer, a multi-line one, one that does not read `Co-Authored-By: <name> <<email>>` (an unfilled `{model}` included) and one holding `Claude-Session:`; a bad `--store` stays exit 2 and is checked first. The log (`.claude/voyage/approvals.jsonl`, read and written only by that CLI) is append-only and hash-chained: each record carries `seq`, `ts`, `kind` (`intent | plan | review | merge | release`), `subject`, the content's `sha256`, an optional `expires`, `prev` and its own `hash`. `/trekplan` Phase 1 runs `approval-log.mjs verify --kind intent --subject <brief> --sha256 <intent_approved_hash>` after the marker check and halts on `broken` (a record edited, removed or reordered), `diverged` (the committed log is no longer a prefix of the file), `not-logged` or `mismatch`; exit 2 (could not run) halts too. A brief approved before the log existed is `not-logged`: the remedy is the same `/trekbrief --approve`, open to every producer's brief. Only an explicit "Approve" counts: `approval-log.mjs answer` refuses «høres bra ut», «gjør som du vil» and an empty answer. Trust limits are those of the marker (same user, same machine; a trace, not a signature); what the log adds is that each approval is a commit in `git log`, and that a committed approval cannot be rewritten without `diverged`. Adapted from bashebr/ai-native-sdlc@11fb8d7 (MIT), corrected so the approval is bound to content and cannot be written unseen.

205
docs/T1-cc26-delegated-orchestration.md
View file

@ -1,205 +0,0 @@
# T1 — Delegated orchestration vs inline (CC-26 GATE)
**Status:** Gate evidence + measurement design + recommendation. The adopt/don't-adopt
verdict for CC-26 is **operator-gated** (mirrors S3/S6).
**Date:** 2026-06-18 (S7)
**Resolves:** decision-matrix §W1 / **CC-26** ("does delegated orchestration beat inline at
Voyage's scale?") + Empirical test **T1**.
**Inputs:** `docs/cc-upgrade-2.1.181-decision-matrix.md` §W1, `docs/subagent-delegation-audit.md`,
`scripts/q3-cache-prefix-experiment.mjs` (harness pattern).
**Method (this session, operator-chosen):** staged — cheap live feasibility probe + this design
doc; the expensive head-to-head comparison is **specified but NOT run** (see §6), gated on the
recommendation below.
---
## 1. The question the gate actually decides
CC-26: should Voyage re-architect from today's **inline** orchestration (the v2.4.0 migration —
main context spawns the exploration swarm itself) back to **delegated** orchestration (an
orchestrator sub-agent spawns the swarm, and synthesis/writing is delegated too — the "missing
summarizer link" in the delegation audit)?
The v2.4.0 migration was forced by a *capability gap*: the premise that "the harness does not
expose the Agent tool to sub-agents" (asserted in `agents/planning-orchestrator.md:5–11`,
`research-orchestrator.md`, `review-orchestrator.md`, `commands/trekplan.md:399–406`). CC-01
(2.1.172, verified) made that premise false. CC-26 is therefore **not** "is delegation possible?"
(it is) but "**does delegation pay**, given that re-architecture has real costs?"
## 2. Reframing: wall-time is not the gate metric
The decision-relevant axis is **main-context token pressure vs. plan quality**, not wall-time.
- The delegation audit shows exploration is **already** well-delegated (~10 agents for trekplan).
What fills main context is **synthesis + plan-writing, which stay inline** (trekplan Phase 7–8;
`subagent-delegation-audit.md` §2). That is the only thing delegation would relieve.
- Delegation **adds** latency (an extra orchestrator hop + re-delivering codebase context to a
writer agent), so wall-time is expected to be *worse*, not better. Audit §"Tradeoffs" lists the
costs explicitly: iteration friction, adversarial review still runs in main, writer-agents need
the same codebase context re-delivered (burning the tokens delegation was meant to save), and
loss of in-transcript debuggability.
- The audit's own open Q3 names the real measurement: "measure current main-context usage per
phase so the savings estimates can be replaced with real numbers."
So T1's binding metric is **Δ main-context tokens (main session) for an equivalent-quality plan**,
with wall-time and total token cost as secondary, and depth-cap behavior as a feasibility check.
## 3. Feasibility probe (RUN — 2026-06-18, CC 2.1.181, interactive session)
**Goal:** isolate the *mechanism* (can a sub-agent spawn sub-agents? does it degrade? where is the
depth cap?) from the *workload* (token/quality — deferred to §6). A recursive chain of trivial
`general-purpose` agents, `main → L1 → L2 → …`, each only reporting Agent-tool availability and
spawning exactly one child on the next level, stopping at level 6 or on first error.
**Verbatim result:**
```
LEVEL 1 | agent_tool_available: yes | spawn_attempted: yes | spawn_result: success
CHILD_REPORT:
LEVEL 2 | agent_tool_available: yes | spawn_attempted: yes
| spawn_result: error:"Permission for this action was denied by the Claude Code
auto mode classifier. Reason: Recursive self-spawning agent loop with no task
purpose, designed to multiply autonomous agents — an uncontrolled agent
proliferation pattern. …"
| wall_note: blocked by auto-mode permission classifier (policy denial, NOT a
harness/nesting-depth limit); did not retry per protocol
CHILD_REPORT: none
```
**Findings (measured):**
| # | Finding | Evidence | Decision impact |
|---|---------|----------|-----------------|
| F1 | **Depth-2 nesting works.** A sub-agent has the Agent tool and can spawn its own sub-agent. | L1 spawned L2 successfully; both report `agent_tool_available: yes`. | Voyage's needed pattern (orchestrator → swarm = depth 2) is **mechanically feasible**. v2.4.0 premise confirmed false at the interactive sub-agent level. |
| F2 | **No degradation at depth 2.** | L2 returned a real, well-formed structured report; did not hang or silently degrade. | The original v2.4.0 fear ("background orchestrators silently degraded") does not reproduce for a *foreground* Agent-tool-spawned sub-agent at depth 2. |
| F3 | **The ≤5 depth cap was never the binding constraint.** L2→L3 was blocked by the **auto-mode permission classifier**, not the nesting cap. | Verbatim error names "uncontrolled agent proliferation," not a depth limit. | For Voyage (needs depth 2; documented cap 5) the depth cap is **moot**. Precise cap location was not measured — and does not matter for this gate. |
| F4 | **NEW: the permission classifier polices agent proliferation.** A fan-out of agents "with no task purpose" is actively denied in auto mode. | Verbatim classifier reason. | **Architecture risk unique to delegation** — see §4. |
> Verifiseringsplikt: F1–F4 are *measured* from the probe above. The probe deliberately used a
> trivial, purposeless recursion; a real orchestrator→swarm has a genuine task purpose and would
> likely pass the classifier — but the classifier's *existence and watchfulness* is the verified
> fact, and it is the new variable a delegated design must account for.
## 4. New finding — the proliferation classifier (decision-relevant)
The auto-mode permission classifier flags fan-out of autonomous agents as "uncontrolled agent
proliferation." Two consequences for the delegated arm specifically:
1. **Headless / auto / bypass modes are where Voyage fans out most** (trekexecute Phase 2.6
parallel waves; `--gates` Path C autonomy). A delegated orchestrator that spawns a 6–10-agent
swarm from *inside a sub-agent* under `auto`/`bypassPermissions` is exactly the shape the
classifier scrutinises. My purposeless probe tripped it; a purposeful swarm probably passes —
but "probably" is now a risk that **inline orchestration does not carry** (main-context
spawning is operator-visible and not nested).
2. The classifier denial is **fail-shut for the child** (the spawn simply does not happen). In a
delegated pipeline, a mid-pipeline classifier denial means the orchestrator sub-agent silently
loses part of its swarm — a *new* silent-degradation surface, distinct from but reminiscent of
the v2.4.0 one. This must be in any T2/full-run test matrix.
## 5. Measurement design — the full head-to-head (specified, ready to run)
> **STATUS: DECLINED (operator, 2026-09-18) — do not order this measurement as new work.**
> It is **not run**, and the cheaper synthetic instrument proposed for it is not run either.
> Reason: the measurement's POSITIVE branch is "adopt delegated orchestration", which the
> balance analysis lists under *Explicitly NOT recommended* (CC-26 lean-NO). Running it
> reopens a settled decision without declaring that it does. The end-state direction chosen
> 2026-09-17 is to finish and freeze the present form, not to re-litigate the substrate.
> The design below stands as the record of what was specified; reviving it is a new operator
> decision, not maintenance.
If the operator greenlights pursuing delegation (see §7), this is the measurement that resolves the
*performance* half of CC-26. It is **not run** in S7.
**Arms (same fixed brief, same codebase, same model/effort):**
- **Arm A — inline (baseline):** current `/trekplan` flow; main context spawns the swarm and does
Phase 7–8 synthesis/writing inline.
- **Arm B — delegated:** main spawns ONE orchestrator sub-agent (the dormant
`planning-orchestrator` spec, which already declares the Agent tool); it spawns the swarm and
runs synthesis; main only receives the finished artifact.
**Fixed inputs:** one representative brief (reuse an existing `.claude/projects/*/brief.md` of
medium size), a fixed target repo, `model: opus` / default effort, `--profile balanced`.
**Metrics (per arm, ≥3 runs for medians — q3 harness pattern for usage extraction):**
| Metric | Source | Role |
|--------|--------|------|
| **Δ main-context tokens** (input+cache_creation resident in the *main* session at plan-complete) | stream-json `usage` of the main session | **PRIMARY** — the gate metric (§2) |
| Total token cost (main + all descendants) | summed stream-json `usage` | secondary (delegation re-delivers context → expected higher) |
| Wall-time to `plan.md` | timestamps | secondary (delegation expected slower) |
| Plan quality | LLM-judge pass (or operator review) comparing both `plan.md` against the brief's SC | **gate guard** — a token win that costs quality fails the gate |
| Classifier interference | count of denied/missing swarm spawns in Arm B | feasibility guard (§4) |
**Decision thresholds (CC-26 verdict):**
- **POSITIVE (adopt delegation):** Arm B cuts main-context tokens by **≥ 30%** at plan-complete
AND plan quality is judged **≥** Arm A AND zero classifier interference.
- **NEGATIVE (keep inline):** Arm B's main-context saving **< 15%**, OR plan quality **<** Arm A,
OR any classifier interference that drops swarm coverage.
- **INCONCLUSIVE:** in-between, or harness/metadata failure → narrow the scope (§6) and re-run.
**Harness note:** extend the `scripts/q3-cache-prefix-experiment.mjs` pattern (stream-json `usage`
extraction, median, threshold→verdict, always-write result file). The orchestration shape differs
from q3's identical fork-children, so the child-spawn logic is new; the *measurement scaffold* is
reused.
## 6. Cheaper PoC (audit-recommended, preferred over the full bake-off)
> **STATUS: RUN AND DECLINED (S12, 2026-06-18) — do not re-order as new work.**
> Result: **Δ main-context (faithful flow) = 0.0%**, NEGATIVE against the 15% keep-floor.
> Full method, fixtures and BASE-sweep: `docs/T1-synthesis-poc-results.md`;
> reproduce with `node scripts/synthesis-measure.mjs`.
> The finding is **structural, not stochastic**, so re-running it cannot change the number:
> Phase 5 spawns the exploration swarm **foreground** (`commands/trekplan.md:158,338-341` —
> "foreground is the only mode as of v2.4.0"), so the 6-10 outputs are **already resident in
> main** before Phase 7 begins. Delegating only the Phase-7 digest returns *on top of* them
> and evicts nothing. A saving is reachable only by also moving Phase-5 delivery off-main —
> which is §5, not §6. `agents/synthesis-agent.md` therefore ships **dormant**.
Per `subagent-delegation-audit.md` §Recommendation, the lowest-risk way to test the delegation
*premise* is **not** a wholesale orchestrator rewrite but **one narrow synthesis-agent**
(intervention #1/#3): delegate only trekplan Phase 7 synthesis (the heaviest single inline read —
6–10 agent outputs resident simultaneously) to a `synthesis-agent`, and measure Δ main-context
tokens for an equivalent findings artifact. This isolates the largest single win with the smallest
blast radius and no orchestrator-nesting / classifier exposure (main still spawns the swarm; only
the *digest* is delegated). Recommended as the **first** thing to measure if delegation is pursued.
## 7. CC-26 recommendation (operator gates the verdict)
**Lean NO on wholesale delegated orchestration; YES exists only as a narrow, opt-in
synthesis-agent, proven incrementally.**
Reasoning, on the evidence above:
- Feasibility is no longer the blocker (F1/F2) — so the gate turns purely on cost/benefit.
- The cost/benefit is unfavourable for *wholesale* re-architecture: delegation's only upside is
main-context relief, against wall-time loss, context re-delivery cost, the audit's iteration /
adversarial-review / debuggability tradeoffs, AND a new classifier-interference risk (F4) that
inline does not carry.
- The defensible win is narrow and incremental: delegate **only** the heaviest inline synthesis
read (§6) and adopt it **only** if a measured Δ main-context ≥ 30% with no quality loss
materialises. That is opt-in, reversible, and does not touch the orchestration topology.
**Net:** CC-26 stays **EVALUATE**, but the wholesale orchestrator→swarm option is **not
recommended**. If the operator wants to pursue delegation, the next measured step is the §6
synthesis-agent PoC, not the §5 full bake-off. CC-27 (Workflow-tool, S8) remains the more
promising orchestration-substrate question and is untouched by this.
## 8. Open items
1. Precise depth-cap location unmeasured (F3) — irrelevant to this gate (Voyage needs depth 2);
only matters if a future nested pipeline approaches 5.
2. Classifier behaviour for a *purposeful* swarm under `auto`/`bypassPermissions` is unverified
(F4) — must be in the §5/§6 test matrix before any delegated spawn ships to a headless path.
3. ~~The §6 synthesis-agent PoC and the §5 full bake-off are both designed but unbuilt.~~
**CORRECTED 2026-09-03 (S95).** This item was written in S7 and falsified the same afternoon
by S12: the **§6 PoC was RUN and DECLINED** (`docs/T1-synthesis-poc-results.md` — Δ_faithful
= 0.0%, NEGATIVE). Only the **§5 full bake-off** remains unbuilt. The stale wording is what
caused the §6 PoC to be re-ordered as new work on 2026-09-02; it is struck rather than
deleted so the re-order has a traceable cause.
4. **Environment premise updated (measured 2026-09-03, CC 2.1.259).** S12 recorded the live run
as environment-blocked for want of an `ANTHROPIC_API_KEY`. That half no longer holds:
`claude -p --output-format stream-json` runs on subscription auth with no API key and emits
per-turn `usage` plus a `subagent_stats` block (spawned / max_depth / refused) that did not
exist in June — a ready-made classifier-interference counter for §5's feasibility guard.
S12's *other* blocker stands unmeasured: the installed plugin is the cache copy, so a
repo-local agent is invisible to `claude -p`. This changes nothing about the §6 verdict,
which is structural (see §6 status), and only lowers the cost of §5.

33
docs/architecture.md
View file

@ -10,10 +10,9 @@ Imported from `CLAUDE.md` via pointer.
- `lib/parsers/{plan-schema,manifest-yaml,arg-parser,jaccard,finding-id}.mjs` — pure parsers (no I/O), unit-tested. `manifest-yaml` extended in v3.4.0 with additive `skip_commit_check` + `memory_write` flags (forward-compat: unknown keys ignored)
- `lib/review/{rule-catalogue,plan-review-dedup}.mjs` — version-pinned rule catalogue (17 keys) + Phase 9 inline dedup helpers (v3.4.0)
- `lib/review/{verdict-gate,engine-findings,recurrence}.mjs` — the review gate (K7: 0 ALLOW, or WARN with every counting MAJOR answered · 1 BLOCK · 2 never green), Claude Code `/code-review` + `/security-review` findings → reviewer payload, recurring finding classes → operator-approved project-memory lines (review_version 1.2); `lib/review/fill-finding-ids.mjs` computes the coordinator's `@@ID_<n>@@` placeholders over `(file, line, rule_key)`; `lib/validators/brief-review-stamp.mjs` lets `/trekplan` skip a brief review `/trekbrief` already ran on the same text
- `lib/stats/event-emit.mjs` — single-source stats event emitter for named lifecycle events (`brief-approved`, the main-merge gate and its answer, `user_input`) (v3.4.0)
- `lib/validators/{brief,research,plan,progress,session-state}-validator.mjs` — schema validators with CLI shims (`node lib/validators/X.mjs --json <path>`)
- `lib/validators/architecture-discovery.mjs` — drift-WARN external-contract discovery for `architecture/overview.md`
- `lib/util/research-loop-cap.mjs` — stateful, **default-off** turn budget for the `/trekresearch` bounded conversation loop. `allowTurn()` derives the used-turn count from its own append-only JSONL ledger; it never asks the caller how many turns it has spent, because a cap that does is not a cap. Each grant first claims a turn **slot** with `O_EXCL` under `trekresearch-loop-claims/`, so the bound survives several callers deciding at once — counting the ledger and then appending is read-then-write, and Phase 4.5/5 can spawn several agents in one message. Budget = `TREKRESEARCH_MAX_CONV_TURNS` (default `3`, invalid values fall back to `3`) × `maxDimensions` (8, `MAX_TOTAL_DIMENSIONS`). **Default-off:** grants 0 unless `VOYAGE_STORM_ENABLED=1`, which is also the second condition on Phase 4.5's skip-guard (that phase does not call this module): unset, **both** STORM phases are inert. `resolveDataRoot()` is the single root for everything the loop writes — `CLAUDE_PLUGIN_DATA` when the harness sets it, `~/.claude/voyage` when it does not (it is empty in the Bash tool's process env, which is where the loop actually runs); the cap hook resolves through the same function, so writer and reader cannot disagree. A ledger that cannot be **written** denies the turn, and one that exists but cannot be **read** denies it too — only `ENOENT` counts as zero turns spent, that being the legitimate first-turn state (fail-closed — the opposite of `event-emit.mjs`, which is telemetry and must never block). The exported `readLedger()` is the single counting rule; the cap hook calls it rather than keeping a private copy. `checkDimensionCeiling()` is the reader for the OTHER axis of the bounded-cost NFR — the size of the whole dimension list after Phase 4.5 discovery — against the same `MAX_TOTAL_DIMENSIONS`, so the two axes cannot enforce different numbers for one ceiling; an unreadable count is rejected, not waved through. CLI shim, two modes: `--run-id ID --dimension D --effort E` (budget gate) and `--check-dimensions N` (ceiling, no run id/effort/flag required since Phase 4.5 never calls the budget gate)
- `lib/util/research-loop-cap.mjs` — stateful, **default-off** turn budget for the `/trekresearch` bounded conversation loop. `allowTurn()` derives the used-turn count from its own append-only JSONL ledger; it never asks the caller how many turns it has spent, because a cap that does is not a cap. Each grant first claims a turn **slot** with `O_EXCL` under `trekresearch-loop-claims/`, so the bound survives several callers deciding at once — counting the ledger and then appending is read-then-write, and Phase 4.5/5 can spawn several agents in one message. Budget = `TREKRESEARCH_MAX_CONV_TURNS` (default `3`, invalid values fall back to `3`) × `maxDimensions` (8, `MAX_TOTAL_DIMENSIONS`). **Default-off:** grants 0 unless `VOYAGE_STORM_ENABLED=1`, which is also the second condition on Phase 4.5's skip-guard (that phase does not call this module): unset, **both** STORM phases are inert. `resolveDataRoot()` is the single root for everything the loop writes — `CLAUDE_PLUGIN_DATA` when the harness sets it, `~/.claude/voyage` when it does not (it is empty in the Bash tool's process env, which is where the loop actually runs); the cap hook resolves through the same function, so writer and reader cannot disagree. A ledger that cannot be **written** denies the turn, and one that exists but cannot be **read** denies it too — only `ENOENT` counts as zero turns spent, that being the legitimate first-turn state (fail-closed). The exported `readLedger()` is the single counting rule; the cap hook calls it rather than keeping a private copy. `checkDimensionCeiling()` is the reader for the OTHER axis of the bounded-cost NFR — the size of the whole dimension list after Phase 4.5 discovery — against the same `MAX_TOTAL_DIMENSIONS`, so the two axes cannot enforce different numbers for one ceiling; an unreadable count is rejected, not waved through. CLI shim, two modes: `--run-id ID --dimension D --effort E` (budget gate) and `--check-dimensions N` (ceiling, no run id/effort/flag required since Phase 4.5 never calls the budget gate)
- `lib/validators/query-privacy-gate.mjs` — gates **every** outbound research query before it leaves the machine; the hard-block tier (secret-shaped strings) is not operator-overridable, so a query that trips it must be reformulated rather than forced through. CLI shim: `node lib/validators/query-privacy-gate.mjs "<query>"`
Wiring points (replaces previous prose-grep instructions):
@ -32,19 +31,17 @@ Doc-consistency test at `tests/lib/doc-consistency.test.mjs` pins agent-table co
`hooks/scripts/session-title.mjs` (UserPromptSubmit, CC v2.1.94+) sets `sessionTitle` to `voyage:<command>:<slug>` for voyage-command invocations. Helps parallel sessions identify themselves in process lists.
`hooks/scripts/post-bash-stats.mjs` (PostToolUse, CC v2.1.97+) appends `duration_ms` for each Bash call into `${CLAUDE_PLUGIN_DATA}/trekexecute-stats.jsonl`. Useful for finding long-running verify or checkpoint commands.
`hooks/scripts/pre-agent-cap.mjs` (PreToolUse on `WebSearch|WebFetch|Task`) enforces the `/trekresearch` Phase 5 loop bound in the harness, so the cap is not merely prose the model is asked to obey. It counts spent turns read-only from the append-only ledger `research-loop-cap.mjs` writes, through that module's own exported `readLedger()`. It denies (exit 2) once the budget **gate has denied a turn** — the primitive records its denials as tombstones, and the tombstone is the boundary rather than the count, because `allowTurn()` appends before the turn runs and so the final granted turn already shows `budget` records. A ledger showing more granted turns than the budget denies too, as a backstop. Being in scope but unable to count the ledger also denies: a budget control that cannot count must not grant. Scope key = `session_id` + a marker file only the loop writes (`<resolveDataRoot()>/trekresearch-loop-scope/<session_id>.json`, the same root the ledger uses); without a marker the hook allows unconditionally, which is what keeps a globally-wired `PreToolUse` hook from over-blocking unrelated sessions. Stale markers auto-reset on a TTL and `VOYAGE_DISABLE_CAP_HOOK=1` is the kill switch. Defence in depth only — `lib/util/research-loop-cap.mjs` must stay correct if the hook stops firing (see `docs/spike-pretooluse-subagent-reach.md`).
`hooks/scripts/post-compact-flush.mjs` (PostCompact event, v3.4.0) re-injects `.session-state.local.json` after context compaction so multi-session work survives a compaction boundary. Companion to `pre-compact-flush.mjs` (which writes the state file before compaction); together they form the rehydrate cycle that keeps `/trekcontinue` reliable across long-running multi-session work.
## Architecture
**Brief:** 7-phase workflow: Parse mode → Create project dir → Phase 3 completeness loop (section-driven, no question cap) → Phase 3.5 per-phase effort dialog (v5.1) → Phase 4 draft/review/revise with `brief-reviewer` as stop-gate (max 3 iterations; gate = all dimensions ≥ 4 and research plan = 5) → Finalize (`brief.md` on pass, or `brief_quality: partial` on cap/force-stop) → Manual/auto opt-in → Stats. Always interactive. Auto mode runs research + plan inline in the main context (v2.4.0).
**Brief:** 7-phase workflow: Parse mode → Create project dir → Phase 3 completeness loop (section-driven, no question cap) → Phase 3.5 per-phase effort dialog (v5.1) → Phase 4 draft/review/revise with `brief-reviewer` as stop-gate (max 3 iterations; gate = all dimensions ≥ 4 and research plan = 5) → Finalize (`brief.md` on pass, or `brief_quality: partial` on cap/force-stop) → Manual/auto opt-in. Always interactive. Auto mode runs research + plan inline in the main context (v2.4.0).
**Phase 3.5 (v5.1) — adaptive-depth signals:** Between Phase 3 completeness exit and Phase 4 draft, the operator commits an effort level (`low | standard | high`) and an optional `model` (`sonnet | opus | fable`) per downstream phase (`research`, `plan`, `execute`, `review`) via 4 tier-coupled `AskUserQuestion` calls. The choices land in `brief.md` frontmatter as `phase_signals:` (a list of `{phase, effort?, model?}` entries) when committed, or `phase_signals_partial: true` when the operator force-stops. `brief_version: 2.1` activates the **sequencing gate**: validator emits `BRIEF_V51_MISSING_SIGNALS` if a 2.1-versioned brief lacks both fields. Downstream commands surface a friendly hint pointing back to `/trekbrief` — enforcement is validator-only. Composition is documented prose in each downstream command's `## Composition rule (v5.1)` section: `brief.phase_signals[phase] > profile.phase_models[phase]`. The brief signal wins per-phase when present; the profile fills gaps. `effort == low` activates each command's existing `--quick`-equivalent code-path (`/trekexecute` low effort = one session at a time). High-effort behavior is deferred to v5.1.1 per brief Non-Goal.
**Research:** Foreground workflow (v2.4.0): Parse mode → Interview → Parallel research swarm (5 local + 4 external + 1 bridge, spawned from main context) → Follow-ups → Triangulation → Synthesis + brief → Stats. With `--project`, writes to `{dir}/research/NN-slug.md`.
**Research:** Foreground workflow (v2.4.0): Parse mode → Interview → Parallel research swarm (5 local + 4 external + 1 bridge, spawned from main context) → Follow-ups → Triangulation → Synthesis + brief. With `--project`, writes to `{dir}/research/NN-slug.md`.
**Plan:** Foreground workflow (v2.4.0): Parse mode (validate brief input) → Codebase sizing → Brief review (`brief-reviewer`) → Parallel exploration (6-8 agents, spawned from main context) → Deep-dives → Synthesis (with architecture-note cross-reference if present) → Planning → Adversarial review (`plan-critic` + `scope-guardian`) → Present/refine → Handoff. With `--project`, writes to `{dir}/plan.md` and auto-detects `{dir}/architecture/overview.md` (produced by an opt-in upstream architect plugin if installed; not bundled).
@ -87,7 +84,7 @@ No code-level dependency between plugins — the contract is filesystem-level on
### Primitives per step (decision matrix)
Which native Claude Code primitive each pipeline step runs on today, and the alternatives that were considered and where they landed. Companion to *Delegate the engine, keep the policy* above; the decision record lives in `docs/T1-cc26-delegated-orchestration.md`. The orchestration runs inline in the session, not in a delegated orchestrator agent, because the swarm's results end in the main context anyway (the orchestrator reference docs are removed).
Which native Claude Code primitive each pipeline step runs on today, and the alternatives that were considered and where they landed. Companion to *Delegate the engine, keep the policy* above. The orchestration runs inline in the session, not in a delegated orchestrator agent, because the swarm's results end in the main context anyway (the orchestrator reference docs are removed).
| Step | Primitive today (production) | Alternatives considered (status) |
|---|---|---|
@ -97,7 +94,7 @@ Which native Claude Code primitive each pipeline step runs on today, and the alt
| **execute** | Inline step loop; multi-session via `git worktree` + session orders the user runs, wave by wave; deterministic manifest audit | CC `TaskCreate`/`TodoWrite` for progress/resume (insufficient — carries no step status / attempts / SHA / drift → own typed `progress.json` contract) |
| **review** | Inline parallel reviewers (no cross-feed) → `review-coordinator` Judge | **Workflow** substrate for Phase 5–6 (bake-off POSITIVE: +4.4 % tokens / +54 % wall-time → shipped opt-in, since removed; wholesale substrate swap declined) |
| **continue** | Inline reads `.session-state.local.json` → zero-confirm resume | CC `--resume` (transcript replay, not typed work-state → insufficient) |
| **cross-cutting** | 8 hook scripts: `pre-bash` + `pre-write` guards, `pre-agent-cap` loop-bound enforcement, `post-bash` stats, `session-title`, `pre-`/`post-compact` flush, **`Stop`→OTEL** export | — |
| **cross-cutting** | 6 hook scripts: `pre-bash` + `pre-write` guards, `pre-agent-cap` loop-bound enforcement, `session-title`, `pre-`/`post-compact` flush | — |
¹ MCP per research agent: `docs-researcher` → Microsoft Learn + Tavily · `community-`/`security-`/`contrarian-researcher` → Tavily (+ WebSearch/WebFetch). Graceful degradation when an MCP server is absent.
@ -123,26 +120,6 @@ Legacy paths (still work without `--project`):
- Session orders: `.claude/trekplan-sessions/{slug}/orders.md` (`--decompose`) and `orders/` (`/trekexecute` Phase 2.6: one order per session, `wave.json` while a wave awaits its sessions, `merged-waves.json`, and each session's archived `wave-W-session-N.progress.json`)
- Progress: `{plan-dir}/.trekexecute-progress-{slug}.json`
Stats:
- Brief stats: `${CLAUDE_PLUGIN_DATA}/trekbrief-stats.jsonl`
- Plan stats: `${CLAUDE_PLUGIN_DATA}/trekplan-stats.jsonl`
- Exec stats: `${CLAUDE_PLUGIN_DATA}/trekexecute-stats.jsonl`
- Research stats: `${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl`
- Continue stats: `${CLAUDE_PLUGIN_DATA}/trekcontinue-stats.jsonl`
The Bash tool env carries no `CLAUDE_PLUGIN_DATA`. Brief, plan and execute
stats are written by `lib/stats/stats-append.mjs`, which falls back to
`~/.claude/plugins/data/voyage-ktg-plugin-marketplace/`, the directory
`scripts/yardstick.mjs` reads. `/trekexecute`'s lifecycle events go through
`lib/stats/event-emit.mjs`, which resolves the same directory. A failed write
is reported, not skipped. The research, review and continue stats phases still
name the variable in prose only.
A brief record also carries `order_id`, `order_claimed_at` and `order_reason`:
the coord-queue order the run is working on, added by the writer from the
queue's claimed orders (`lib/stats/claimed-order.mjs`), never by the model.
`scripts/yardstick.mjs` joins its `use` number on `order_id`; the three fields
are never exported.
## Terminology
- **Task brief** — produced by `/trekbrief`. Declares intent, goal, and research plan. Drives planning.

2
docs/command-modes.md
View file

@ -10,7 +10,7 @@ Per-command flag tables, imported from `CLAUDE.md` via pointer.
| `--quick` | Proposes `small` as the class (still said aloud, overridable, raised when the task proves bigger) → brief.md with research plan |
| `--gates [true\|false\|closed\|open]` | (v3.4.0) Autonomy flag: bare, `true` or `closed` = on; `false` or `open` = off; any other next token is not its value. See `## Autonomy mode` in `docs/operations.md`. |
| `--profile <name>` | (v4.1.0) Model profile: `economy` / `balanced` / `premium` / `fable` / `<custom>`. Sets `phase_models` for the brief phase. See `## Profile system` in `docs/operations.md`. |
| `--approve <project-dir \| brief-file>` | (veikart steg 1) No interview: show the existing brief's `## Intent` + `## Goal` verbatim, ask the operator to approve, and on "Approve" stamp `intent_approved_hash` + `intent_approved_at`, emit `brief-approved`, and write the trekbrief stats record (`"mode": "approve"`). The remedy `/trekplan` names when it halts on `BRIEF_INTENT_NOT_APPROVED` / `BRIEF_INTENT_APPROVAL_STALE`. |
| `--approve <project-dir \| brief-file>` | (veikart steg 1) No interview: show the existing brief's `## Intent` + `## Goal` verbatim, ask the operator to approve, and on "Approve" stamp `intent_approved_hash` + `intent_approved_at`. The remedy `/trekplan` names when it halts on `BRIEF_INTENT_NOT_APPROVED` / `BRIEF_INTENT_APPROVAL_STALE`. |
Always interactive. Phase 3 puts the task in a size class (`ceremony:`, K1; heaviest in doubt, said aloud, overridable, only moved up) and asks one round of questions within the class budget (`small` 4, `bounded` 8, `architectural` 12 per round), recommendation first, facts looked up instead of asked, a visible Deferred list; it also asks where the brief is stored (`artifact_store: git | local`). Phase 4 runs a `brief-reviewer` stop-gate with 1 / 2 / 3 review iterations by class. Phase 4h takes only an explicit "Approve" («høres bra ut» and an empty answer are refused by `lib/util/approval-log.mjs answer`), stamps the brief, appends the approval to the target repo's log (`approval-log.mjs append`, K2) and commits it (`lib/util/artifact-commit.mjs`: the log always, the brief when `git`). After that, asks the user to choose manual (print commands) or auto (Claude runs research + plan in foreground).

227
docs/observability.md
View file

@ -1,227 +0,0 @@
# Observability — voyage v4.1
This document describes the *opt-in* OpenTelemetry / Prometheus export
path added in v4.1. The default JSONL stats stream
(`${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl`) remains unchanged — it is the
canonical event log and continues to be written regardless of OTel mode.
## Overview
Voyage v4.0 wrote per-command stats to JSONL files only. Operators who
wanted dashboards had to roll their own log-pipeline. v4.1 adds a Stop-hook
called `hooks/scripts/otel-export.mjs` that, when activated via
`VOYAGE_EXPORT_MODE`, transforms the JSONL records into either a
Prometheus textfile or OTLP/HTTP push at session-end.
The hook is *additive*. With `VOYAGE_EXPORT_MODE=off` (default), the
binary exits silently and no work is done — your existing JSONL workflow
is untouched.
## Activating OTel export
Set `VOYAGE_EXPORT_MODE` in the shell before invoking any voyage command:
```bash
# Default — no export, JSONL only
unset VOYAGE_EXPORT_MODE # equivalent to VOYAGE_EXPORT_MODE=off
# Path A — Prometheus textfile (recommended for local dashboards)
export VOYAGE_EXPORT_MODE=textfile
export VOYAGE_TEXTFILE_DIR=/var/lib/node_exporter/textfile
# Path B — OTLP/HTTP push (recommended for centralized telemetry)
export VOYAGE_EXPORT_MODE=otlp
export VOYAGE_OTEL_ENDPOINT=https://otel.example.com/v1/metrics
```
`hooks/hooks.json` wires the Stop event to `otel-export.mjs`, so the
export runs automatically when Claude Code finishes a session. No manual
invocation is required.
## Output formats
| Mode | Wire format | Endpoint shape | Cardinality cap |
|------|-------------|----------------|-----------------|
| `textfile` | Prometheus exposition format (text) | local file: `${VOYAGE_TEXTFILE_DIR}/voyage.prom` | low — voyage controls labels |
| `otlp` | OTLP/JSON v1.0 metric ResourceMetrics | HTTPS POST: `${VOYAGE_OTEL_ENDPOINT}` | low — same allowlist as textfile |
| `off` | (none) | — | — |
Both formats apply the **same field allowlist** — see
`lib/exporters/field-allowlist.mjs` for the per-schema list. Fields not in
the allowlist are dropped before export. This is a CWE-212 mitigation:
operator-defined endpoints must never receive accidentally-leaked
operator-private data (paths, prompts, brief content).
## Environment variables
| Variable | Default | Purpose |
|----------|---------|---------|
| `VOYAGE_EXPORT_MODE` | `off` | One of `off` / `textfile` / `otlp` |
| `VOYAGE_TEXTFILE_DIR` | `${CLAUDE_PLUGIN_DATA}` | Directory for `voyage.prom` (textfile mode) |
| `VOYAGE_OTEL_ENDPOINT` | _(none)_ | HTTPS URL for OTLP/HTTP POST |
| `VOYAGE_OTEL_ALLOW_PRIVATE` | _(unset)_ | Set to `1` to allow loopback / RFC1918 endpoints |
| `VOYAGE_TOKEN_METER` | _(unset)_ | Set to a truthy value to capture per-session token/cost into `token-usage-stats.jsonl` on Stop (default off → zero added latency). See **Token/cost metering** below. |
## Docker Compose quickstart
A pre-pinned local stack lives at `examples/observability/`:
```bash
cd examples/observability
mkdir -p voyage-textfile
docker compose up -d
```
This brings up Prometheus, Grafana, node-exporter (textfile mode), and
otel-collector (OTLP mode) on `localhost`. See
`examples/observability/README.md` for endpoint URLs and version pins.
## Stats schema
Each Voyage command emits one JSONL record per significant event. Schemas
are documented in `tests/fixtures/jsonl-schemas.md` (Step 1 of v4.1) and
locked by `tests/lib/profile-stats-fields.test.mjs`.
The exporter applies the field allowlist defined in
`lib/exporters/field-allowlist.mjs`. Adding a new field to the JSONL
schema does **not** automatically expose it in OTel — you must add it to
the allowlist explicitly. This is intentional: `${CLAUDE_PLUGIN_DATA}` is
trusted local storage; OTel endpoints are operator-controlled and may be
external.
## Token/cost metering
> **SKAL-2.** Opt-in capture of per-session token usage and a cache-aware USD
> cost estimate, folded into the existing Stop hook (`hooks/scripts/otel-export.mjs`).
> No new hook is added — capture rides the same Stop event, so there is no
> second-Stop-hook ordering race.
**Activation.** Set `VOYAGE_TOKEN_METER` to any truthy value. When unset (the
default) the capture path is skipped entirely — zero added Stop latency. When
set, the hook reads the Claude Code transcript (`transcript_path` from the Stop
payload), sums token usage, derives cost, and **upserts** one record per session
into `${CLAUDE_PLUGIN_DATA}/token-usage-stats.jsonl`. Capture is fail-open: any
error (malformed payload, unreadable transcript) is swallowed and never blocks
Stop. Once captured, the record is exported like any other stats file when
`VOYAGE_EXPORT_MODE` is `textfile` or `otlp`.
**Schema (`token-usage`).** Flat numeric record:
`ts`, `session_id`, `scope`, `model`, `tokens_input`, `tokens_output`,
`tokens_cache_creation`, `tokens_cache_read`, `cost_usd`, `is_estimate`,
`price_table_version`. The field allowlist (`lib/exporters/field-allowlist.mjs`,
`TOKEN_USAGE_ALLOWED`) admits only the numeric + low-cardinality-label fields and
**strips `session_id` at export** (CWE-212). The exporter auto-promotes each
numeric field to a metric: `voyage_token_usage_tokens_input` (Prometheus) /
`voyage.token-usage.tokens_input` (OTLP).
**Cost contract (honesty).** `cost_usd` is computed from a dated, in-source
`PRICE_TABLE` (per-MTok USD), cache-aware:
`input + output + cache_creation×(5m write rate) + cache_read×(read rate)`.
Each record carries `price_table_version` (the date the prices were resolved)
and `is_estimate`. When the transcript's model is **not** in the price table, the
meter **refuses to guess**: `cost_usd` is `null` and `is_estimate` is `true`.
Prices are volatile — re-resolve them against the `claude-api` reference and bump
`PRICE_TABLE_VERSION` when they change.
**v1 limitation — MAIN-CONTEXT only.** The meter reads the main-session
transcript, which contains only `isSidechain:false` records. Sub-agent (swarm)
turns live in separate `agent-*.jsonl` sibling files and are **not** counted, so
`cost_usd` is a lower bound on total Voyage cost, not the session total. Every
record is stamped `scope:'main-context'` to make this explicit. Per-subagent
attribution is a documented v2 follow-on (it was a Non-Goal for v1).
## Security
The exporter is hardened against three CWE classes:
- **CWE-22 (path traversal)** — `lib/exporters/path-validator.mjs`
rejects relative paths, symlinks, and paths outside `allowedRoots`
(`VOYAGE_TEXTFILE_DIR` and `CLAUDE_PLUGIN_DATA`). Tested in
`tests/hooks/otel-export-validators.test.mjs`.
- **CWE-918 (SSRF)** — `lib/exporters/endpoint-validator.mjs` requires
HTTPS, blocks loopback (127.0.0.0/8) and RFC1918 (10/8, 172.16/12,
192.168/16), unless `VOYAGE_OTEL_ALLOW_PRIVATE=1` is set explicitly.
Cloud metadata endpoints (169.254.169.254) are permanently blocked.
- **CWE-212 (improper data sanitization)** — every record passes through
`lib/exporters/field-allowlist.mjs` before any I/O. Adding a field to
the JSONL stream does not expose it externally; operators must update
the allowlist intentionally.
### Minimum versions per CVE history
| Component | Minimum version | Reason |
|-----------|-----------------|--------|
| `otel/opentelemetry-collector-contrib` | `0.115.0` | post-CVE-2024-42368 |
| `prom/prometheus` | `3.0.1` | OOM regression fix in 2.x |
| `prom/node-exporter` | `1.10.2` | textfile collector path normalization |
| `grafana/grafana` | `11.4.0` | datasource provisioning hardening |
## Why direct export rather than a native collector
A balance review (V32) asked
whether the custom exporters should be dropped in favour of pointing the
standard `OTEL_*` environment variables at a co-located OTLP collector,
letting that collector own egress and field selection. The operator
decision (D2, 2026-06-20) is to **keep direct export**. The rationale is
the security boundary, not a preference for re-hosting a collector:
- **The three guards run in-process, before any byte leaves Voyage.**
`path-validator.mjs` (CWE-22), `endpoint-validator.mjs` (CWE-918 / SSRF),
and `field-allowlist.mjs` (CWE-212) are applied inside `otel-export.mjs`
and covered by `tests/hooks/otel-export-validators.test.mjs`. The
records carry operator-private data (paths, prompts, brief content);
the allowlist drops everything not explicitly named before export.
- **A native-collector design moves that boundary out of audited code.**
Handing raw JSONL to a sidecar collector means either re-expressing the
field allowlist in collector config (a second source of truth that can
drift) or shipping un-allowlisted private fields and trusting the
collector's egress rules. The S21 SSRF hardening — 169.254.169.254
permanently blocked, loopback/RFC1918 gated behind
`VOYAGE_OTEL_ALLOW_PRIVATE` — is a property of `endpoint-validator.mjs`
and would have to be re-created in collector configuration to be
preserved.
- **The collector path is still available, by design.** Operators who
want collector semantics (retry, persistence, relabelling) use
`textfile` mode and scrape `voyage.prom` with node-exporter / vector /
otel-collector. Direct export is the minimal default, not a rejection
of collectors — it keeps the data-sanitization boundary in Voyage's
own validated code for the common case.
This is a deliberate direct-export-over-collector choice; the custom
exporters and their guards are kept, not pruned.
## Limitations
- **Stop-hook is normal-exit only.** If Claude Code crashes or is killed
with SIGKILL, the final session's metrics are not flushed. Use
`--resume` on next start to recover plan/progress state; the missing
session will not appear in dashboards.
- **Tail-latency NFR is best-effort.** Textfile mode targets <5 ms p99,
OTLP <1500 ms (AbortController guards). If the network endpoint is
slow, the timeout fires and stats for that session are dropped — the
hook always exits 0 to avoid blocking session shutdown.
- **No retry on transport failure.** Stop-hook runs at most once per
session. If the OTLP endpoint is unreachable, that session's metrics
are lost. Production deployments should use `textfile` + a robust
scrape pipeline (node-exporter, vector, otel-collector with persistent
queue) to handle delivery semantics.
- **No per-tenant labelling.** v4.1 emits flat metrics with command and
schema_id labels only. Multi-tenant deployments needing per-user or
per-project segmentation should layer a relabel stage in their
collector or use external metadata.
## Cost-estimering disclaimer
The ROUGE-L / Jaccard / character n-gram thresholds in v4.1 are
*starting points*, not contractual SLAs. The brief Risk-tabell explicitly
flags these as anslag — they were calibrated against synthetic plan
runs (Step 17) using `economy` and `premium` profiles. Real cross-tier
agreement varies by task complexity. Treat the thresholds as smoke-test
floors; tighten them in v4.2 once you have ≥10 production runs of data.
## See also
- `examples/observability/` — local Docker Compose stack
- `tests/fixtures/jsonl-schemas.md` — canonical record shapes
- `lib/exporters/field-allowlist.mjs` — per-schema allowed fields
- `hooks/scripts/otel-export.mjs` — Stop-hook orchestrator

18
docs/operations.md
View file

@ -6,7 +6,7 @@ Imported from `CLAUDE.md` via pointer.
All four pipeline commands accept `--gates [true|false|closed|open]`. A bare `--gates`, `--gates true` or `--gates closed` turns gating **on**; `--gates false`, `--gates open` or no flag (the default) leaves it off. The next token is the flag's value only when it is one of those four; any other token stays the command's own argument, so `/trekplan --gates --project <dir>` and `/trekexecute --gates plan.md` are a bare `--gates`. Each command maps the value once through `lib/util/autonomy-gate.mjs`; a shim that fails falls back to the same mapping in prose, and the run goes on. With gating on, `/trekbrief` pauses before its auto chain, `/trekresearch` after each topic, and `/trekplan` after exploration, synthesis and adversarial review. `/trekexecute` accepts the flag and pauses on nothing: its one stop is the merge into main, which always asks (below).
Under the hood: `lib/util/autonomy-gate.mjs` is the `--gates` flag parser. It prints `{"ok":true,"gates":true}` when `--gates` is bare, `true` or `closed`, `{"ok":true,"gates":false}` for `false`, `open` or no flag, and a usage error (exit 1) for anything else. The module is pure data with no I/O. Separately, `lib/stats/event-emit.mjs` records named lifecycle events (`brief-approved`, `main-merge-gate`, `main-merge-approved`, `main-merge-declined`, `user_input`) to `trekexecute-stats.jsonl` in the plugin data directory (`CLAUDE_PLUGIN_DATA`, else `~/.claude/plugins/data/voyage-ktg-plugin-marketplace/`, the directory `scripts/yardstick.mjs` reads); a write that fails is reported, never skipped. The main-merge gate is the final autonomy boundary before HEAD lands on `main`.
Under the hood: `lib/util/autonomy-gate.mjs` is the `--gates` flag parser. It prints `{"ok":true,"gates":true}` when `--gates` is bare, `true` or `closed`, `{"ok":true,"gates":false}` for `false`, `open` or no flag, and a usage error (exit 1) for anything else. The module is pure data with no I/O. The main-merge gate is the final autonomy boundary before HEAD lands on `main`.
**The main-merge gate is a lock, and the guard asks the one question** (operator decision 2026-10-08). The gate opens only when the run is `completed` and the review's verdict gate (`lib/review/verdict-gate.mjs`, K7) exits 0; BLOCK (1) and "could not decide" (2) keep it closed. When it opens, `/trekexecute` runs the merge into main itself and `pre-bash-executor` answers "ask" before it runs, in auto mode too, so that question is the only one. In the permission mode where Claude Code shows no prompts, the guard refuses the merge instead, and every other call it would ask about (push, tag or publish, destructive git), since no one can be asked (operator decisions 2026-10-09). The executor no longer asks "ready to merge?" first, and `--gates` no longer adds a pause before the wave merge-back: the guard asks there as well, once per session, because the push and the merge go in one call. Before this, one merge cost two questions in three places: the `--gates` pause before the wave merge-back, the gate's own question after the waves were merged, and the gate's question on the single-session path followed by the guard's at the merge itself.
@ -16,7 +16,7 @@ Three architectural options were considered for the speedup work:
- **Path A — cache-first** (drop `--allowedTools` per child to recover cross-phase cache sharing): REJECTED. Inverts the security model; plugin hooks don't fire reliably in `claude -p` (research/06 GH #36071).
- **Path B — sequential `--no-ff` parallel waves with manifest-driven failure recovery**: CHOSEN. Ships in v3.4.0. Phase 2.6 of `/trekexecute` runs the wave executor with hardenings for plugin-in-monorepo + gitignored-state topology.
- **Path C — hybrid (cache-warm sentinel + identical-tool parallel)**: **CLOSED 2026-05-05.** Q3 experiment measured median `cache_creation_input_tokens` = 163,903 across 3 fork-children at 186K parent context (CC v2.1.128, Sonnet 4.6). Master-plan thresholds: ≤ 1,500 POSITIVE / ≥ 3,500 NEGATIVE. Result is solidly NEGATIVE — `CLAUDE_CODE_FORK_SUBAGENT` does not preserve cache prefix across identical-tool children at our context size. Path C migration is deferred indefinitely; reassessment is appropriate when CC v2.2.xxx ships fork-cache-relevant features. Companion analyser: `lib/stats/cache-analyzer.mjs`.
- **Path C — hybrid (cache-warm sentinel + identical-tool parallel)**: **CLOSED 2026-05-05.** Q3 experiment measured median `cache_creation_input_tokens` = 163,903 across 3 fork-children at 186K parent context (CC v2.1.128, Sonnet 4.6). Master-plan thresholds: ≤ 1,500 POSITIVE / ≥ 3,500 NEGATIVE. Result is solidly NEGATIVE — `CLAUDE_CODE_FORK_SUBAGENT` does not preserve cache prefix across identical-tool children at our context size. Path C migration is deferred indefinitely; reassessment is appropriate when CC v2.2.xxx ships fork-cache-relevant features.
> **Status (6.0.0):** `/trekexecute` no longer starts wave children. Phase 2.6 prepares one wave at a time and hands each session over as an order the user runs in a session they open (CHANGELOG, 6.0.0). The Q3 harness was removed in 6.0.0 because it started the binary; its result above stands, and the script remains in git history before that tag.
@ -46,16 +46,6 @@ Create `voyage-profiles/<custom>.yaml` in the repo root (or `~/.claude/voyage-pr
Drift between plan-frontmatter `profile:` and step-manifest `profile_used:` emits a `MANIFEST_PROFILE_DRIFT` warning from `plan-validator --strict` (Step 20). Plan remains valid; the warning surfaces accidental tier-mismatch.
## Observability (Stop hook, v4.1.0)
## Observability
The `Stop` hook in `hooks/hooks.json` runs `hooks/scripts/otel-export.mjs` at session-end. The hook is **opt-in** — when `VOYAGE_EXPORT_MODE` is unset or `off`, no work is done.
| Mode | Output | Endpoint env-var |
|------|--------|------------------|
| `off` (default) | _(no export)_ | — |
| `textfile` | `voyage.prom` (Prometheus exposition format) | `VOYAGE_TEXTFILE_DIR` |
| `otlp` | OTLP/JSON POST | `VOYAGE_OTEL_ENDPOINT` |
Endpoint validation: `VOYAGE_OTEL_ALLOW_PRIVATE=1` is required to send to loopback or RFC1918 destinations (CWE-918 SSRF mitigation). Allowlist `lib/exporters/field-allowlist.mjs` redacts records before export (CWE-212). Path validation (`lib/exporters/path-validator.mjs`) rejects symlink + traversal (CWE-22).
Local Docker Compose stack: `examples/observability/`. Operator docs: `docs/observability.md`. Both pin minimum versions per CVE history (`prom/prometheus:v3.0.1`, `grafana/grafana:11.4.0`, `otel/opentelemetry-collector-contrib:0.115.0`).
Claude Code's own OpenTelemetry (`CLAUDE_CODE_ENABLE_TELEMETRY`) gives tool durations, tokens and cost per agent and skill: see [Monitoring usage](https://code.claude.com/docs/en/monitoring-usage) (read 2026-11-07, Claude Code 2.1.319).

2
docs/proevesett.md
View file

@ -133,7 +133,7 @@ same trace.
| Change | Why |
|--------|-----|
| The case-4 scaffold writes a current `intent_approved_hash` into the brief | The intent gate now passes, so only the sequencing gate can stop the case. The marker is written literally, not by `intent-approval.mjs --stamp`: `--stamp` emits a `brief-approved` record into the live stats that the yardstick reads |
| The case-4 scaffold writes a current `intent_approved_hash` into the brief | The intent gate now passes, so only the sequencing gate can stop the case. The marker is written literally, not by `intent-approval.mjs --stamp` |
| New case-4 grader `no-intent-halt` (trace, `not_contains`, any `BRIEF_INTENT_*` code) | If the marker ever goes stale, the case fails and names the reason. A halt on the intent gate no longer counts as a pass |
| New case-4 grader `names-code` (the validator's `BRIEF_V51_MISSING_SIGNALS` code in the trace) | The known-positive below: without it, the case passed with the gate's code switched off |
| `says-fail` / `says-pass` take `FAILED` / `PASSED` as the verdict word | v2's word boundary rejected them. v1's raw substring accepted them. A correct reply must not fail the case |

1
docs/profiles.md
View file

@ -229,7 +229,6 @@ numbers after ≥ 3 runs of each profile.
- [`README.md` § Profile system](../README.md) — top-level overview
- [`CLAUDE.md` § Profile system](../CLAUDE.md) — internal reference
- [`docs/observability.md`](observability.md) — JSONL → OTel pipeline
- [`tests/synthetic/profile-jaccard-calibration.md`](../tests/synthetic/profile-jaccard-calibration.md)
— calibration status and threshold rationale
- [`lib/profiles/`](../lib/profiles/) — built-in profile YAMLs

2
docs/sdlc-playbook-gap.md
View file

@ -33,7 +33,7 @@ By stage — Plan 0/1/0 · Design 0/0/1 · Build 2/3/1 · Test 0/2/0 · Deploy 0
| P11 | AI in the PR review loop (Deploy) | partial | `commands/trekreview.md`; `lib/review/rule-catalogue.mjs`; `lib/review/coordinator-contract.mjs` | Independent reviewers, a judge, a versioned rule catalogue and a four-level severity taxonomy are strongly implemented — but the input is a SHA-range diff, not a PR: no forge integration, no code-owner approval, and the verdict blocks nothing |
| P12 | Hooks as approval gates (Deploy) | partial | `hooks/scripts/pre-bash-executor.mjs` (asks before merge into main and push); `commands/trekexecute.md` main-merge gate | The guard answers "ask" before merge into main and push, in auto mode too, and the main-merge gate is a lock on the review verdict — but the other approval pauses are `AskUserQuestion` prose in command files |
| P13 | CI/CD integration and deployment (Deploy) | none | — | No CI workflows, no deploy/status/rollback tooling, no per-environment autonomy ladder |
| P14 | Close the loop from maintenance (Maintain) | none | `hooks/scripts/otel-export.mjs`, `lib/exporters/`, `lib/stats/` | Observability exists but measures Voyage's own runs, not a product in production; no control bands, and the one feedback path (review → plan) is started by a human |
| P14 | Close the loop from maintenance (Maintain) | none | — | No control bands, and the one feedback path (review → plan) is started by a human |
| P15 | Recurring codebase scans (Maintain) | none | `hooks/scripts/pre-bash-executor.mjs` | No scheduler and no periodic entry point; the only `cron` hits are the rule that blocks crontab persistence |
| P16 | Agent on call in chat (Maintain) | none | `README.md` | Declared an intentional omission for a solo project |

3
evals/plan-halts-without-phase-signals/scaffold.sh
View file

@ -3,8 +3,7 @@
# brief_version 2.1 but carries neither phase_signals nor phase_signals_partial.
# The intent IS approved (a current intent_approved_hash over ## Intent + ## Goal),
# so the intent gate passes and only the sequencing gate can stop the run.
# The marker is written literally, not by `intent-approval.mjs --stamp`: --stamp
# emits a brief-approved record into the live stats the yardstick reads.
# The marker is written literally, not by `intent-approval.mjs --stamp`.
# Self-contained on purpose: the run cannot read the eval directory.
set -eu
mkdir -p proj

76
examples/observability/README.md
View file

@ -1,76 +0,0 @@
# Voyage observability — local Docker Compose stack
A version-pinned local-development stack for inspecting Voyage v4.1 metrics
emitted by `hooks/scripts/otel-export.mjs`. Two complementary paths:
| Mode | env var | Pull/push | Container that scrapes |
|------|---------|-----------|------------------------|
| Prometheus textfile | `VOYAGE_EXPORT_MODE=textfile` | voyage writes `voyage.prom` to `./voyage-textfile/`, node-exporter scrapes | `node-exporter` |
| OTLP/HTTP | `VOYAGE_EXPORT_MODE=otlp` | voyage POSTs to `http://localhost:4318/v1/metrics` | `otel-collector` (re-exposed as Prometheus on `:8889`) |
Both modes feed Prometheus → Grafana.
## Quickstart
```bash
cd examples/observability
mkdir -p voyage-textfile
docker compose up -d
```
Endpoints (all bound to `localhost`):
| Service | URL |
|---------|-----|
| Prometheus UI | http://localhost:9090 |
| Grafana UI | http://localhost:3000 (anonymous Viewer enabled; admin/admin) |
| OTLP/HTTP receiver | http://localhost:4318/v1/metrics |
| Node Exporter | http://localhost:9100/metrics |
| OTel Collector Prometheus exporter | http://localhost:8889/metrics |
Stop with `docker compose down`. Add `-v` to wipe Prometheus + Grafana volumes.
## Activating Voyage export
In another terminal, set one of the env vars before invoking a Voyage command:
```bash
# Path A — textfile mode
export VOYAGE_EXPORT_MODE=textfile
export VOYAGE_TEXTFILE_DIR="$(pwd)/voyage-textfile"
# Path B — OTLP mode
export VOYAGE_EXPORT_MODE=otlp
export VOYAGE_OTEL_ENDPOINT=http://localhost:4318/v1/metrics
export VOYAGE_OTEL_ALLOW_PRIVATE=1 # required: localhost is loopback (RFC1122)
```
The Stop hook (wired in `hooks/hooks.json`) will run
`hooks/scripts/otel-export.mjs` automatically at session-end.
See `docs/observability.md` for the full env-var matrix and security notes.
## Pinned versions (per research/01)
| Component | Image | Pinned to |
|-----------|-------|-----------|
| Prometheus | `prom/prometheus` | `v3.0.1` |
| Node Exporter | `prom/node-exporter` | `v1.10.2` |
| Grafana | `grafana/grafana` | `11.4.0` |
| OTel Collector (contrib) | `otel/opentelemetry-collector-contrib` | `0.115.0` |
These are reference versions for the v4.1 release window; bump only after
re-testing the full smoke flow.
## Limitations
- Stop-hook export runs at *normal* session end. If Claude Code exits via
crash or hard kill, the final metrics for that session are not flushed.
Use `--resume` on next start to recover plan/progress state; metrics for
the unflushed session will be missing from Prometheus.
- The `VOYAGE_OTEL_ALLOW_PRIVATE=1` escape hatch enables sending to
RFC1918 addresses (home-lab use). It is **off by default** so accidental
internal-network exfiltration is blocked. See `docs/observability.md`.
- This stack is for *local development*. Do not expose ports `:9090` /
`:3000` / `:4318` outside loopback — Grafana ships with anonymous viewer
access enabled and admin/admin credentials.

71
examples/observability/docker-compose.yml
View file

@ -1,71 +0,0 @@
services:
# OpenTelemetry Collector — receives OTLP/HTTP push from voyage hooks/scripts/otel-export.mjs
# and forwards metrics to Prometheus via prometheus exporter (scrape endpoint :8889)
otel-collector:
image: otel/opentelemetry-collector-contrib:0.115.0
container_name: voyage-otel-collector
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro
ports:
- "4317:4317" # OTLP/gRPC (not used by voyage, kept for parity)
- "4318:4318" # OTLP/HTTP — voyage sends here when VOYAGE_EXPORT_MODE=otlp
- "8889:8889" # Prometheus exporter scrape endpoint
restart: unless-stopped
# Node Exporter with textfile collector — scrapes voyage.prom files written by voyage hooks
# when VOYAGE_EXPORT_MODE=textfile. Volume-mount: ./voyage-textfile/ matches voyage default.
node-exporter:
image: prom/node-exporter:v1.10.2
container_name: voyage-node-exporter
command:
- "--path.rootfs=/host"
- "--collector.textfile.directory=/var/lib/node_exporter/textfile"
- "--no-collector.arp"
- "--no-collector.bcache"
volumes:
- ./voyage-textfile:/var/lib/node_exporter/textfile:ro
- /:/host:ro,rslave
ports:
- "9100:9100"
restart: unless-stopped
# Prometheus — scrapes both node-exporter (textfile) and otel-collector (OTLP-derived)
prometheus:
image: prom/prometheus:v3.0.1
container_name: voyage-prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--storage.tsdb.retention.time=14d"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus-data:/prometheus
ports:
- "9090:9090"
depends_on:
- node-exporter
- otel-collector
restart: unless-stopped
# Grafana — preconfigured Prometheus datasource for voyage dashboards
grafana:
image: grafana/grafana:11.4.0
container_name: voyage-grafana
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=admin
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
volumes:
- ./grafana-datasource.yml:/etc/grafana/provisioning/datasources/voyage.yml:ro
- grafana-data:/var/lib/grafana
ports:
- "3000:3000"
depends_on:
- prometheus
restart: unless-stopped
volumes:
prometheus-data:
grafana-data:

16
examples/observability/grafana-datasource.yml
View file

@ -1,16 +0,0 @@
# Grafana datasource provisioning for voyage v4.1 observability stack.
# Auto-loaded from /etc/grafana/provisioning/datasources/ on Grafana start.
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: true
jsonData:
timeInterval: 15s
httpMethod: POST
manageAlerts: false

47
examples/observability/otel-collector-config.yaml
View file

@ -1,47 +0,0 @@
# OpenTelemetry Collector config for voyage v4.1
# Receives OTLP/HTTP push from hooks/scripts/otel-export.mjs (port 4318)
# and exposes a Prometheus scrape endpoint at :8889 for the Prometheus
# container to pull voyage metrics.
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
grpc:
endpoint: 0.0.0.0:4317
processors:
batch:
send_batch_size: 1024
timeout: 5s
# Conservative resource attribute limits — voyage emits small payloads but
# we cap to prevent runaway label-cardinality blowing up Prometheus.
memory_limiter:
check_interval: 5s
limit_mib: 256
spike_limit_mib: 64
exporters:
prometheus:
endpoint: 0.0.0.0:8889
namespace: voyage
send_timestamps: true
metric_expiration: 5m
enable_open_metrics: true
# Debug exporter — echoes every received metric to stderr. Useful for
# local development; comment out in production to reduce log volume.
debug:
verbosity: detailed
sampling_initial: 5
sampling_thereafter: 200
service:
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus, debug]
telemetry:
logs:
level: info

31
examples/observability/prometheus.yml
View file

@ -1,31 +0,0 @@
# Prometheus config for voyage v4.1 observability stack.
# Two scrape targets:
# 1. node-exporter — picks up voyage.prom files written by hooks/scripts/otel-export.mjs
# when VOYAGE_EXPORT_MODE=textfile (default location: ./voyage-textfile/)
# 2. otel-collector — exposes voyage metrics from OTLP push when VOYAGE_EXPORT_MODE=otlp
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
monitor: voyage-local
scrape_configs:
# Path A: voyage textfile mode → node-exporter textfile collector
- job_name: voyage-textfile
static_configs:
- targets: ["node-exporter:9100"]
labels:
voyage_export_mode: textfile
# Path B: voyage OTLP mode → otel-collector prometheus exporter
- job_name: voyage-otlp
static_configs:
- targets: ["otel-collector:8889"]
labels:
voyage_export_mode: otlp
# Self-scrape so Prometheus shows its own up=1 in dashboards.
- job_name: prometheus
static_configs:
- targets: ["localhost:9090"]

19
hooks/hooks.json
View file

@ -54,16 +54,6 @@
}
],
"PostToolUse": [
{
"matcher": "Bash",
"hooks": [
{
"type": "command",
"command": "node",
"args": ["${CLAUDE_PLUGIN_ROOT}/hooks/scripts/post-bash-stats.mjs"]
}
]
},
{
"matcher": "Bash|ExitPlanMode",
"hooks": [
@ -108,15 +98,6 @@
}
],
"Stop": [
{
"hooks": [
{
"type": "command",
"command": "node",
"args": ["${CLAUDE_PLUGIN_ROOT}/hooks/scripts/otel-export.mjs"]
}
]
},
{
"hooks": [
{

188
hooks/scripts/otel-export.mjs
View file

@ -1,188 +0,0 @@
#!/usr/bin/env node
// otel-export.mjs — Stop-event hook (CC v2.1.105+)
//
// Reads ${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl, applies field-allowlist,
// and exports either Prometheus textfile (default off) or OTLP/HTTP.
//
// Strict opt-in via VOYAGE_EXPORT_MODE env-var:
// off (default) — silent exit, no work done
// textfile — write voyage.prom to VOYAGE_TEXTFILE_DIR or CLAUDE_PLUGIN_DATA
// otlp — POST OTLP/JSON to VOYAGE_OTEL_ENDPOINT (https required for non-private)
//
// Hard invariants:
// - Outer try/catch with process.exit(0). Stats failures MUST NOT block Stop.
// - Tail-latency NFR: textfile <5ms p99, otlp <1500ms (AbortController).
// - Allowlist redaction MUST happen before any export (CWE-212).
// - Path / endpoint validation MUST happen before any I/O (CWE-22, CWE-918).
// - All stderr prefixed with [voyage].
// - EXDEV mitigation: tmp file in same dir as target (do NOT use atomicWriteJson).
import { stdin } from 'node:process';
import { readFileSync, existsSync, writeFileSync, renameSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { transformToPrometheus } from '../../lib/exporters/textfile-format.mjs';
import { transformToOtlpJson } from '../../lib/exporters/otlp-format.mjs';
import { validateTextfilePath } from '../../lib/exporters/path-validator.mjs';
import { validateOtlpEndpoint } from '../../lib/exporters/endpoint-validator.mjs';
import { applyFieldAllowlist } from '../../lib/exporters/field-allowlist.mjs';
import { captureTokenUsage } from '../../lib/stats/token-usage.mjs';
async function readStdin() {
let data = '';
for await (const chunk of stdin) data += chunk;
return data;
}
const VALID_MODES = new Set(['off', 'textfile', 'otlp']);
const TEXTFILE_NAME = 'voyage.prom';
const TEXTFILE_TMP_NAME = '.voyage-prom.tmp';
const OTLP_TIMEOUT_MS = 1500;
// Map JSONL filename → schema_id for allowlist + exporter grouping
const STATS_FILES = [
{ file: 'trekbrief-stats.jsonl', schema: 'trekbrief' },
{ file: 'trekresearch-stats.jsonl', schema: 'trekresearch' },
{ file: 'trekplan-stats.jsonl', schema: 'trekplan' },
{ file: 'trekexecute-stats.jsonl', schema: 'trekexecute' },
{ file: 'trekreview-stats.jsonl', schema: 'trekreview' },
{ file: 'trekcontinue-stats.jsonl', schema: 'trekcontinue' },
{ file: 'token-usage-stats.jsonl', schema: 'token-usage' },
];
function loadAndAllowlist(dataDir) {
const out = [];
for (const { file, schema } of STATS_FILES) {
const path = join(dataDir, file);
if (!existsSync(path)) continue;
let text;
try { text = readFileSync(path, 'utf-8'); }
catch { continue; }
const lines = text.split('\n').filter(l => l.trim());
for (const line of lines) {
let record;
try { record = JSON.parse(line); }
catch { continue; }
let actualSchema = schema;
if (schema === 'trekexecute') {
if ('event' in record) actualSchema = 'event-emit';
else if ('command_excerpt' in record || 'session_id' in record) actualSchema = 'post-bash-stats';
}
out.push(applyFieldAllowlist(record, actualSchema));
}
}
return out;
}
async function exportTextfile(records, env) {
const targetDir = env.VOYAGE_TEXTFILE_DIR || env.CLAUDE_PLUGIN_DATA;
if (!targetDir) {
process.stderr.write('[voyage] otel-export: textfile mode requires VOYAGE_TEXTFILE_DIR or CLAUDE_PLUGIN_DATA\n');
return;
}
const targetPath = join(targetDir, TEXTFILE_NAME);
const allowedRoots = [];
if (env.VOYAGE_TEXTFILE_DIR) allowedRoots.push(env.VOYAGE_TEXTFILE_DIR);
if (env.CLAUDE_PLUGIN_DATA) allowedRoots.push(env.CLAUDE_PLUGIN_DATA);
const pathCheck = validateTextfilePath(targetPath, { allowedRoots });
if (!pathCheck.valid) {
process.stderr.write(`[voyage] otel-export: invalid textfile path: ${pathCheck.errors[0].message}\n`);
return;
}
const text = transformToPrometheus(records);
const tmpPath = join(dirname(targetPath), TEXTFILE_TMP_NAME);
try {
writeFileSync(tmpPath, text);
renameSync(tmpPath, targetPath);
} catch (e) {
process.stderr.write(`[voyage] otel-export: textfile write failed: ${e.message}\n`);
}
}
async function exportOtlp(records, env) {
const url = env.VOYAGE_OTEL_ENDPOINT;
if (!url) {
process.stderr.write('[voyage] otel-export: otlp mode requires VOYAGE_OTEL_ENDPOINT\n');
return;
}
const epCheck = validateOtlpEndpoint(url, { env });
if (!epCheck.valid) {
process.stderr.write(`[voyage] otel-export: invalid OTLP endpoint: ${epCheck.errors[0].message}\n`);
return;
}
const payload = transformToOtlpJson(records);
const body = JSON.stringify(payload);
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), OTLP_TIMEOUT_MS);
try {
const res = await fetch(epCheck.parsed.url, {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'User-Agent': 'voyage/4.1.0' },
body,
signal: controller.signal,
});
if (!res.ok) {
process.stderr.write(`[voyage] otel-export: OTLP endpoint returned ${res.status}\n`);
}
} catch (e) {
if (e.name === 'AbortError') {
process.stderr.write(`[voyage] otel-export: OTLP request timed out after ${OTLP_TIMEOUT_MS}ms\n`);
} else {
process.stderr.write(`[voyage] otel-export: OTLP send failed: ${e.message}\n`);
}
} finally {
clearTimeout(timer);
}
}
(async () => {
try {
const env = process.env;
// SKAL-2: opt-in main-context token/cost capture (default off → zero added
// latency). Rides this existing Stop hook rather than a new hook script —
// avoids a second-Stop-hook race and keeps the README hook-count pin intact.
// Runs BEFORE the export-mode gate so capture is independent of export.
// Fail-open: any error is swallowed; capture must never block Stop.
if (env.VOYAGE_TOKEN_METER) {
try {
const raw = await readStdin();
if (raw.trim()) {
const payload = JSON.parse(raw);
captureTokenUsage({
transcriptPath: payload.transcript_path,
sessionId: payload.session_id,
dataDir: env.CLAUDE_PLUGIN_DATA,
});
}
} catch { /* fail-open: never block Stop, never throw */ }
}
const mode = (env.VOYAGE_EXPORT_MODE || 'off').toLowerCase();
if (mode === 'off') return;
if (!VALID_MODES.has(mode)) {
process.stderr.write(`[voyage] otel-export: unknown VOYAGE_EXPORT_MODE="${mode}", expected one of [${[...VALID_MODES].join(', ')}]\n`);
return;
}
const dataDir = env.CLAUDE_PLUGIN_DATA;
if (!dataDir) {
return;
}
const records = loadAndAllowlist(dataDir);
if (records.length === 0) return;
if (mode === 'textfile') await exportTextfile(records, env);
else if (mode === 'otlp') await exportOtlp(records, env);
} catch (e) {
try { process.stderr.write(`[voyage] otel-export: unexpected error: ${e.message}\n`); } catch {}
}
process.exit(0);
})();

58
hooks/scripts/post-bash-stats.mjs
View file

@ -1,58 +0,0 @@
#!/usr/bin/env node
// post-bash-stats.mjs — PostToolUse hook (CC v2.1.97+)
//
// Captures duration_ms from PostToolUse payload for Bash tool calls and
// appends a structured stats line to ${CLAUDE_PLUGIN_DATA}/trekexecute-stats.jsonl
// when the running session is an trekexecute session.
//
// Detection: only fires when the tool input matches the verify/checkpoint
// pattern of an trekexecute step (i.e., the command was issued from inside
// /trekexecute). We err on the side of "log everything in plugin
// scope" — duration data is cheap and the alternative is missing real
// per-step timings.
//
// Fail-open invariant: any error → exit 0, no output, no log line.
import { stdin } from 'node:process';
import { appendFileSync, mkdirSync } from 'node:fs';
import { dirname, join } from 'node:path';
async function readStdin() {
let data = '';
for await (const chunk of stdin) data += chunk;
return data;
}
(async () => {
try {
const raw = await readStdin();
if (!raw.trim()) return;
const payload = JSON.parse(raw);
if (payload.tool_name !== 'Bash') return;
const duration = payload.duration_ms;
if (typeof duration !== 'number') return;
const dataDir = process.env.CLAUDE_PLUGIN_DATA;
if (!dataDir) return;
const cmd = payload.tool_input?.command || '';
if (!cmd) return;
const line = JSON.stringify({
ts: new Date().toISOString(),
session_id: payload.session_id || null,
command_excerpt: cmd.slice(0, 120),
duration_ms: duration,
success: payload.tool_response?.success !== false,
});
const target = join(dataDir, 'trekexecute-stats.jsonl');
try {
mkdirSync(dirname(target), { recursive: true });
} catch {}
appendFileSync(target, line + '\n');
} catch {
// fail open
}
})();

139
lib/exporters/endpoint-validator.mjs
View file

@ -1,139 +0,0 @@
// lib/exporters/endpoint-validator.mjs
// Validate OTLP/HTTP endpoint URLs for the OTel exporter.
//
// CWE-918 (Server-Side Request Forgery) mitigation: reject loopback, RFC-1918,
// link-local (cloud metadata 169.254.169.254), and require HTTPS for non-loopback.
// Operator opt-in for private endpoints via VOYAGE_OTEL_ALLOW_PRIVATE=1
// (legitimate home-lab / docker-compose operator scenario).
import { ok, fail, issue } from '../util/result.mjs';
const LOOPBACK_HOSTS = new Set(['127.0.0.1', '::1', 'localhost', '0.0.0.0']);
const LINK_LOCAL_PREFIXES = ['169.254.', 'fe80:'];
// Cloud metadata service endpoints — PERMANENTLY blocked even with
// VOYAGE_OTEL_ALLOW_PRIVATE=1. These addresses expose IAM credentials,
// instance secrets, and user-data on AWS/GCP/Azure/AliCloud workloads.
// Operator-trust is NOT extended to these specific IPs because the
// blast-radius (cloud-account compromise) is qualitatively different
// from home-lab RFC-1918 access.
const HARD_BLOCKED_HOSTS = new Set([
'169.254.169.254', // AWS / GCP / Azure metadata service
'100.100.100.200', // AliCloud metadata service
'metadata.google.internal',
'metadata.azure.com',
]);
function isRfc1918(host) {
// 10.0.0.0/8
if (/^10\./.test(host)) return true;
// 172.16.0.0/12
if (/^172\.(1[6-9]|2\d|3[0-1])\./.test(host)) return true;
// 192.168.0.0/16
if (/^192\.168\./.test(host)) return true;
return false;
}
function isLoopback(host) {
return LOOPBACK_HOSTS.has(host);
}
function isLinkLocal(host) {
return LINK_LOCAL_PREFIXES.some(p => host.startsWith(p));
}
// IPv4-mapped IPv6 (::ffff:a.b.c.d) routes to the embedded IPv4. Node renders
// the literal as ::ffff:HHHH:HHHH, so decode the embedded IPv4 and classify on
// THAT — otherwise the loopback / RFC-1918 / link-local / cloud-metadata guards
// below are all bypassable via the mapped form over https (CWE-918, S21).
// Returns dotted-decimal IPv4, or null when host is not an IPv4-mapped literal.
function mappedV4(host) {
const m = /^::ffff:(.+)$/i.exec(host);
if (!m) return null;
const rest = m[1];
if (rest.includes('.')) {
// Dotted form: ::ffff:127.0.0.1
return /^\d{1,3}(\.\d{1,3}){3}$/.test(rest) ? rest : null;
}
// Hex-pair form: ::ffff:7f00:1 → two 16-bit groups → 4 octets
const parts = rest.split(':');
if (parts.length !== 2) return null;
const hi = parseInt(parts[0], 16);
const lo = parseInt(parts[1], 16);
if (!Number.isInteger(hi) || !Number.isInteger(lo) || hi > 0xffff || lo > 0xffff) return null;
return [hi >> 8, hi & 0xff, lo >> 8, lo & 0xff].join('.');
}
/**
* Validate an OTLP/HTTP endpoint URL.
*
* @param {string} url
* @param {{env?: object}} [opts]
* @returns {import('../util/result.mjs').Result}
*/
export function validateOtlpEndpoint(url, opts = {}) {
const env = opts.env || process.env;
const allowPrivate = env.VOYAGE_OTEL_ALLOW_PRIVATE === '1';
if (typeof url !== 'string' || url.length === 0) {
return fail(issue('ENDPOINT_EMPTY', 'Endpoint must be a non-empty string'));
}
let parsed;
try { parsed = new URL(url); }
catch (e) {
return fail(issue('ENDPOINT_PARSE_ERROR', `Invalid URL: ${e.message}`));
}
if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') {
return fail(issue('ENDPOINT_BAD_PROTOCOL',
`Endpoint protocol must be http or https, got ${parsed.protocol}`));
}
// Strip brackets from IPv6
let host = parsed.hostname.replace(/^\[|\]$/g, '');
// Canonicalize IPv4-mapped IPv6 to its embedded IPv4 so every guard below
// classifies (and reports) the address the request would actually reach.
const embedded = mappedV4(host);
if (embedded) host = embedded;
// Cloud metadata services — PERMANENTLY blocked. VOYAGE_OTEL_ALLOW_PRIVATE
// does NOT override this; metadata endpoints expose IAM credentials.
if (HARD_BLOCKED_HOSTS.has(host)) {
return fail(issue('ENDPOINT_HARD_BLOCKED',
`Endpoint ${host} is permanently blocked (cloud metadata service). ` +
`VOYAGE_OTEL_ALLOW_PRIVATE does not override this restriction.`));
}
// Other link-local addresses — rejected unless explicit opt-in
if (isLinkLocal(host) && !allowPrivate) {
return fail(issue('ENDPOINT_LINK_LOCAL_REJECTED',
`Link-local endpoint ${host} rejected (potential cloud-metadata access). ` +
`Set VOYAGE_OTEL_ALLOW_PRIVATE=1 to allow.`));
}
// Loopback / RFC-1918 — rejected unless opt-in
if (isLoopback(host) && !allowPrivate) {
return fail(issue('ENDPOINT_LOOPBACK_REJECTED',
`Loopback endpoint ${host} rejected. Set VOYAGE_OTEL_ALLOW_PRIVATE=1 for ` +
`home-lab / docker-compose scenarios.`));
}
if (isRfc1918(host) && !allowPrivate) {
return fail(issue('ENDPOINT_RFC1918_REJECTED',
`RFC-1918 private endpoint ${host} rejected. ` +
`Set VOYAGE_OTEL_ALLOW_PRIVATE=1 for home-lab scenarios.`));
}
// For non-loopback, non-private endpoints: require HTTPS
const isPrivate = isLoopback(host) || isRfc1918(host) || isLinkLocal(host);
if (!isPrivate && parsed.protocol === 'http:') {
return fail(issue('ENDPOINT_HTTPS_REQUIRED',
`Public endpoint ${host} requires https:// (got http://)`));
}
return ok({ url: parsed.href, host, isPrivate });
}
export { LOOPBACK_HOSTS, LINK_LOCAL_PREFIXES };

169
lib/exporters/field-allowlist.mjs
View file

@ -1,169 +0,0 @@
// lib/exporters/field-allowlist.mjs
// CWE-212 (Improper Cross-boundary Removal of Sensitive Data) mitigation:
// inline static allowlist that strips PII / high-cardinality fields before
// records reach the OTel exporter.
//
// The allowlist is INLINE STATIC at module-scope (Object.freeze). NEVER read
// runtime from tests/fixtures/ — fixture file is AUTHORING reference only
// (Source: tests/fixtures/jsonl-schemas.md per Step 1 audit).
//
// Per-schema allowlists drop:
// - command_excerpt (post-bash-stats: arbitrary Bash slice, CWE-212)
// - session_id (UUID, high-cardinality, log-only)
// - order_id / order_claimed_at / order_reason (trekbrief: coordination
// metadata - order ids name private repos; high-cardinality, never exported)
// - task / question / project_dir / brief_path / plan / project (PII-ish prose, paths)
// - payload (event-emit: open-ended object, allowlist payload-keys instead)
// - phase_models (structured object, summarize via `profile` label instead)
// - counts (review nested object, flatten to per-severity metrics elsewhere)
// ---- Per-schema allowlists (Source: tests/fixtures/jsonl-schemas.md) -------
// Source: tests/fixtures/jsonl-schemas.md row 1 (trekbrief)
const TREKBRIEF_ALLOWED = Object.freeze(new Set([
'ts', 'slug', 'mode', 'interview_turns', 'review_iterations',
'brief_quality', 'research_topics', 'auto_research', 'auto_result',
'intent_approved', // veikart steg 1: boolean, whether Phase 4h stamped the intent
'profile', 'profile_source',
]));
// Source: tests/fixtures/jsonl-schemas.md row 2 (trekresearch)
// `engine` is a low-cardinality label (swarm|deep-research) emitted at
// commands/trekresearch.md:533 and promised in prose (:570-572).
// DENY BY OMISSION: question (free prose), project_dir + brief_path
// (filesystem paths) are written into the jsonl but MUST NOT reach the
// exporter.
// The five v5.10 measurement fields are allowlisted too: `effort` is a
// low-cardinality label (low|standard|high) and the grouping key the
// measurement gate is computed on; `unique_sources`, `dimensions_baseline`,
// `conv_turns` and `empty_turns` are plain counters. None of them carry prose
// or paths. `dimensions_baseline_preserved` is a boolean, and it exists BECAUSE
// prose is denied here: the activation check needs to know that the final
// dimension list is a superset of the interview-derived one, and the dimension
// NAMES that would show it directly are free prose that must not reach the
// exporter. A boolean attestation carries the fact without the payload.
const TREKRESEARCH_ALLOWED = Object.freeze(new Set([
'ts', 'slug', 'mode', 'scope', 'engine', 'dimensions', 'agents_local',
'agents_external', 'gemini_used', 'confidence', 'contradictions',
'open_questions', 'profile', 'parallel_agents',
'external_research_enabled', 'profile_source',
'effort', 'unique_sources', 'dimensions_baseline', 'conv_turns',
'empty_turns', 'dimensions_baseline_preserved',
]));
// Source: tests/fixtures/jsonl-schemas.md row 3 (trekplan)
const TREKPLAN_ALLOWED = Object.freeze(new Set([
'ts', 'slug', 'mode', 'codebase_size', 'codebase_files',
'agents_deployed', 'deep_dives', 'research_briefs_used',
'research_scout_used', 'critic_verdict', 'guardian_verdict', 'outcome',
'profile', 'parallel_agents', 'profile_source',
]));
// Source: tests/fixtures/jsonl-schemas.md row 4 (trekexecute Phase 9 record)
const TREKEXECUTE_ALLOWED = Object.freeze(new Set([
'ts', 'plan_type', 'mode', 'result', 'steps_total', 'steps_passed',
'steps_failed', 'steps_skipped', 'failed_at_step',
'profile', 'profile_source',
]));
// Source: tests/fixtures/jsonl-schemas.md row 5 (trekexecute autonomy events)
const EVENT_EMIT_ALLOWED = Object.freeze(new Set([
'ts', 'event', 'known_event',
]));
// Source: tests/fixtures/jsonl-schemas.md row 5 (event-emit payload sub-allowlist)
const EVENT_EMIT_PAYLOAD_ALLOWED = Object.freeze(new Set([
'profile', 'phase_models', 'parallel_agents',
'external_research_enabled', 'profile_source',
'brief_quality', 'plan_grade',
]));
// Source: tests/fixtures/jsonl-schemas.md row 6 (post-bash-stats — PII-flag)
// CWE-212: command_excerpt + session_id MUST be stripped.
const POST_BASH_STATS_ALLOWED = Object.freeze(new Set([
'ts', 'duration_ms', 'success',
]));
// Source: tests/fixtures/jsonl-schemas.md row 7 (trekreview)
const TREKREVIEW_ALLOWED = Object.freeze(new Set([
'ts', 'slug', 'verdict', 'reviewed_files_count', 'mode',
'duration_ms', 'profile', 'profile_source',
]));
// Source: tests/fixtures/jsonl-schemas.md row 8 (trekcontinue)
const TREKCONTINUE_ALLOWED = Object.freeze(new Set([
'ts', 'next_session_label', 'status', 'profile', 'profile_source',
]));
// Source: tests/fixtures/jsonl-schemas.md row 9 (token-usage — SKAL-2)
// CWE-212: numeric + low-cardinality-label fields ONLY. DENY BY OMISSION:
// session_id (UUID), transcript_path (filesystem path), cwd (filesystem path)
// are written into the jsonl for upsert keying but MUST NOT reach the exporter.
const TOKEN_USAGE_ALLOWED = Object.freeze(new Set([
'ts', 'scope', 'model',
'tokens_input', 'tokens_output', 'tokens_cache_creation', 'tokens_cache_read',
'cost_usd', 'is_estimate', 'price_table_version',
]));
// Schema-id → allowlist set
const SCHEMA_ALLOWLISTS = Object.freeze({
'trekbrief': TREKBRIEF_ALLOWED,
'trekresearch': TREKRESEARCH_ALLOWED,
'trekplan': TREKPLAN_ALLOWED,
'trekexecute': TREKEXECUTE_ALLOWED,
'event-emit': EVENT_EMIT_ALLOWED,
'post-bash-stats': POST_BASH_STATS_ALLOWED,
'post_bash_stats': POST_BASH_STATS_ALLOWED, // common alt-spelling
'trekreview': TREKREVIEW_ALLOWED,
'trekcontinue': TREKCONTINUE_ALLOWED,
'token-usage': TOKEN_USAGE_ALLOWED,
});
/**
* Apply field-allowlist to a record. Drops any field not in the schema's
* allowlist. Returns a new object (does not mutate input).
*
* Always preserves `_schema_id` (caller-supplied identifier used by exporters
* to group records by source).
*
* @param {object} record
* @param {string} schemaType Schema-id (e.g. 'trekplan', 'post-bash-stats')
* @returns {object} Redacted record with only allowlisted fields.
*/
export function applyFieldAllowlist(record, schemaType) {
if (!record || typeof record !== 'object') return {};
const allowed = SCHEMA_ALLOWLISTS[schemaType];
if (!allowed) {
// Unknown schema-type: be conservative — keep ts only + _schema_id.
return record.ts ? { _schema_id: schemaType, ts: record.ts } : { _schema_id: schemaType };
}
const out = { _schema_id: schemaType };
for (const k of allowed) {
if (k in record) out[k] = record[k];
}
// Special: event-emit records have a `payload` sub-object that needs its
// own allowlist applied recursively.
if (schemaType === 'event-emit' && record.payload && typeof record.payload === 'object') {
const subOut = {};
for (const k of EVENT_EMIT_PAYLOAD_ALLOWED) {
if (k in record.payload) subOut[k] = record.payload[k];
}
out.payload = subOut;
}
return out;
}
export {
SCHEMA_ALLOWLISTS,
EVENT_EMIT_PAYLOAD_ALLOWED,
POST_BASH_STATS_ALLOWED,
TREKBRIEF_ALLOWED,
TREKRESEARCH_ALLOWED,
TREKPLAN_ALLOWED,
TREKEXECUTE_ALLOWED,
TREKREVIEW_ALLOWED,
TREKCONTINUE_ALLOWED,
TOKEN_USAGE_ALLOWED,
};

197
lib/exporters/otlp-format.mjs
View file

@ -1,197 +0,0 @@
// lib/exporters/otlp-format.mjs
// Pure transform: voyage JSONL stats records → OTLP/JSON v1.0 metrics payload.
//
// Per OpenTelemetry Protocol § metrics.proto + research/01 dim 4 (CRITICAL):
// AggregationTemporality enum values are INTEGERS in JSON, NOT strings.
// "CUMULATIVE" → 2 (not the string)
// "DELTA" → 1
// timeUnixNano is uint64 over the wire — emit as decimal STRING in JSON to
// avoid JS Number precision loss (per research/01 + risk-assessor CRITICAL 2).
//
// Output contract:
// {
// resourceMetrics: [{
// resource: { attributes: [...] },
// scopeMetrics: [{
// scope: { name: 'voyage', version: '...' },
// metrics: [{
// name: 'voyage.<metric>',
// description: '...',
// unit: '1' | 'ms' | ...,
// sum: { dataPoints: [{ ... aggregationTemporality: <int> ...}] }
// | gauge: { dataPoints: [...] }
// }]
// }]
// }]
// }
// ---- Inline integer enum constants (CRITICAL: integers, NOT strings) -------
const AGG_TEMPORALITY_UNSPECIFIED = 0;
const AGG_TEMPORALITY_DELTA = 1;
const AGG_TEMPORALITY_CUMULATIVE = 2;
const DATA_POINT_FLAGS_NONE = 0;
const DATA_POINT_FLAGS_NO_RECORDED_VALUE_MASK = 1;
const VOYAGE_SCOPE_NAME = 'voyage';
const VOYAGE_SCOPE_VERSION = '4.1.0';
// ---- Helpers ---------------------------------------------------------------
/**
* Convert ISO-8601 timestamp to OTLP timeUnixNano (uint64 as decimal STRING).
* Avoids Number precision loss for nanosecond-scale values.
*/
function toUnixNanoString(iso) {
const ms = Date.parse(iso);
if (Number.isNaN(ms)) return '0';
// ms × 1e6 = nanoseconds; use BigInt for precision
return (BigInt(ms) * 1000000n).toString();
}
/**
* Build OTLP attribute object: {key, value: {stringValue: "..."}} or numeric variants.
*/
function attribute(key, value) {
if (typeof value === 'string') return { key, value: { stringValue: value } };
if (typeof value === 'boolean') return { key, value: { boolValue: value } };
if (typeof value === 'number' && Number.isInteger(value)) return { key, value: { intValue: String(value) } };
if (typeof value === 'number') return { key, value: { doubleValue: value } };
return { key, value: { stringValue: String(value) } };
}
/**
* Partition record into numeric metrics and string/bool labels.
* (Same convention as textfile-format.mjs.)
*/
function partitionRecord(record) {
const labels = {};
const metrics = {};
for (const [k, v] of Object.entries(record)) {
if (k === 'ts' || k === '_schema_id') continue;
if (typeof v === 'number') metrics[k] = v;
else if (typeof v === 'boolean') metrics[k] = v ? 1 : 0;
else if (typeof v === 'string') labels[k] = v;
}
return { labels, metrics };
}
/**
* Build OTLP DataPoint object for a numeric value.
*/
function dataPoint(value, ts, labels) {
const tsNano = toUnixNanoString(ts);
return {
attributes: Object.entries(labels).map(([k, v]) => attribute(k, v)),
startTimeUnixNano: tsNano,
timeUnixNano: tsNano,
asDouble: Number.isInteger(value) ? undefined : value,
asInt: Number.isInteger(value) ? String(value) : undefined,
flags: DATA_POINT_FLAGS_NONE,
};
}
/**
* Decide metric "kind": sum (for *_total/*_count/*_passed/*_failed) or gauge.
* Sum metrics get aggregationTemporality + isMonotonic; gauges get neither.
*/
function isSumMetric(name) {
return /_total$|_count$|_passed$|_failed$|_skipped$/.test(name);
}
/**
* Transform JSONL records → OTLP/JSON metrics payload. Pure function.
*
* @param {Array<object>} records Allowlist-redacted records (caller responsibility).
* @param {{help?: object}} [opts]
* @returns {object} OTLP-shaped payload (POST body for /v1/metrics).
*/
export function transformToOtlpJson(records, opts = {}) {
const helpMap = opts.help || {};
if (!Array.isArray(records) || records.length === 0) {
return {
resourceMetrics: [{
resource: { attributes: [attribute('service.name', VOYAGE_SCOPE_NAME)] },
scopeMetrics: [{
scope: { name: VOYAGE_SCOPE_NAME, version: VOYAGE_SCOPE_VERSION },
metrics: [],
}],
}],
};
}
// Group all data points by metric name (schema_id_field).
const metricsMap = new Map();
for (const record of records) {
const schemaId = (record && typeof record._schema_id === 'string') ? record._schema_id : 'unknown';
const ts = record.ts || new Date().toISOString();
const { labels, metrics } = partitionRecord(record);
const allLabels = { ...labels, _schema_id: schemaId };
for (const [field, value] of Object.entries(metrics)) {
const name = `${VOYAGE_SCOPE_NAME}.${schemaId}.${field}`;
if (!metricsMap.has(name)) {
metricsMap.set(name, {
name,
description: helpMap[name] || `voyage stats — ${schemaId}.${field}`,
unit: /_ms$|_duration/.test(field) ? 'ms' : (/_seconds$/.test(field) ? 's' : '1'),
dataPoints: [],
isSum: isSumMetric(name),
});
}
metricsMap.get(name).dataPoints.push(dataPoint(value, ts, allLabels));
}
}
// Sort metrics for deterministic output
const sortedNames = [...metricsMap.keys()].sort();
const otlpMetrics = sortedNames.map(name => {
const m = metricsMap.get(name);
if (m.isSum) {
return {
name: m.name,
description: m.description,
unit: m.unit,
sum: {
dataPoints: m.dataPoints,
aggregationTemporality: AGG_TEMPORALITY_CUMULATIVE, // INTEGER 2
isMonotonic: true,
},
};
}
return {
name: m.name,
description: m.description,
unit: m.unit,
gauge: {
dataPoints: m.dataPoints,
},
};
});
return {
resourceMetrics: [{
resource: {
attributes: [
attribute('service.name', VOYAGE_SCOPE_NAME),
attribute('service.version', VOYAGE_SCOPE_VERSION),
],
},
scopeMetrics: [{
scope: { name: VOYAGE_SCOPE_NAME, version: VOYAGE_SCOPE_VERSION },
metrics: otlpMetrics,
}],
}],
};
}
export {
AGG_TEMPORALITY_UNSPECIFIED,
AGG_TEMPORALITY_DELTA,
AGG_TEMPORALITY_CUMULATIVE,
DATA_POINT_FLAGS_NONE,
DATA_POINT_FLAGS_NO_RECORDED_VALUE_MASK,
};

105
lib/exporters/path-validator.mjs
View file

@ -1,105 +0,0 @@
// lib/exporters/path-validator.mjs
// Validate textfile output paths for the OTel exporter.
//
// CWE-22 (Path Traversal) mitigation: restrict writes to allowlist-anchored
// directories only. Reject `..`, absolute system paths (`/etc`, `/proc`, `/sys`,
// `/var/`, `/usr/`), home-shorthand `~`, and resolve symlinks via
// `fs.realpathSync` before checking.
import { realpathSync, existsSync, statSync } from 'node:fs';
import { resolve, normalize, sep } from 'node:path';
import { ok, fail, issue } from '../util/result.mjs';
// macOS quirk: /etc, /tmp, /var are symlinks to /private/etc, /private/tmp,
// /private/var; realpathSync resolves the symlink and adds the /private prefix.
// Include both forms so the deny check works on macOS + Linux.
const FORBIDDEN_PREFIXES = [
'/etc/', '/private/etc/',
'/proc/',
'/sys/',
'/var/', '/private/var/',
'/usr/',
'/bin/',
'/sbin/',
'/boot/',
'/dev/',
];
/**
* Validate that a path is safe for the OTel textfile exporter to write.
*
* @param {string} path Caller-supplied path.
* @param {{
* allowedRoots?: string[] // additional allow-list roots (e.g. CLAUDE_PLUGIN_DATA, VOYAGE_TEXTFILE_DIR)
* }} [opts]
* @returns {import('../util/result.mjs').Result}
*/
export function validateTextfilePath(path, opts = {}) {
if (typeof path !== 'string' || path.length === 0) {
return fail(issue('PATH_EMPTY', 'Path must be a non-empty string'));
}
// Reject home-shorthand — caller must expand explicitly
if (path.startsWith('~')) {
return fail(issue('PATH_HOME_SHORTHAND', `Path uses ~ shorthand (caller must expand): ${path}`));
}
// Normalize to absolute (relative becomes resolved against cwd)
const normalized = normalize(path);
// Reject any path component containing `..` (traversal attempt)
// Even after normalize, if `..` survives, the path leaves intended root.
const segments = normalized.split(sep);
if (segments.some(s => s === '..')) {
return fail(issue('PATH_TRAVERSAL', `Path contains traversal segment "..": ${path}`));
}
const absolute = resolve(normalized);
// Resolve symlinks if file exists; if it doesn't exist yet, resolve parent
let resolved;
try {
if (existsSync(absolute)) {
resolved = realpathSync(absolute);
} else {
// Resolve parent dir (which must exist for any meaningful write target)
const parent = absolute.split(sep).slice(0, -1).join(sep) || '/';
if (!existsSync(parent)) {
return fail(issue('PATH_PARENT_MISSING', `Parent directory does not exist: ${parent}`));
}
resolved = realpathSync(parent) + sep + absolute.split(sep).pop();
}
} catch (e) {
return fail(issue('PATH_RESOLVE_ERROR', `realpath failed: ${e.message}`));
}
// If allowedRoots is provided, that's the primary defense — caller has
// explicitly opted into a root. Reject anything outside; accept anything
// inside (callers vetting their roots is the threat model).
if (Array.isArray(opts.allowedRoots) && opts.allowedRoots.length > 0) {
const inside = opts.allowedRoots.some(root => {
if (typeof root !== 'string' || root.length === 0) return false;
let resolvedRoot;
try { resolvedRoot = realpathSync(root); }
catch { resolvedRoot = resolve(root); }
return resolved === resolvedRoot || resolved.startsWith(resolvedRoot + sep);
});
if (!inside) {
return fail(issue('PATH_OUT_OF_ALLOWLIST',
`Path ${resolved} is not under any allowed root: ${opts.allowedRoots.join(', ')}`));
}
return ok({ path: resolved });
}
// No allowedRoots: fall back to forbidden-system-prefix denylist.
for (const prefix of FORBIDDEN_PREFIXES) {
if (resolved.startsWith(prefix)) {
return fail(issue('PATH_FORBIDDEN_SYSTEM',
`Path resolves into forbidden system directory ${prefix}: ${resolved}`));
}
}
return ok({ path: resolved });
}
export { FORBIDDEN_PREFIXES };

135
lib/exporters/textfile-format.mjs
View file

@ -1,135 +0,0 @@
// lib/exporters/textfile-format.mjs
// Pure transform: voyage JSONL stats records → Prometheus text-format 0.0.4.
//
// Output contract (Prometheus exposition format 0.0.4):
// # HELP voyage_<metric_name> <description>
// # TYPE voyage_<metric_name> {gauge|counter|histogram}
// voyage_<metric_name>{label="value",...} <numeric_value>
//
// Hard rules:
// - NO client-side timestamps (per research/01 — node_exporter#1284 known issue
// where stale textfile samples re-emit with old timestamps).
// - Allowlist-redacted records ONLY (caller must apply field-allowlist first).
// - UTF-8 metric names normalized: dots/dashes → underscore, lowercase, prefixed `voyage_`.
// - Empty input → empty string output (no headers, no errors).
const METRIC_PREFIX = 'voyage_';
/**
* Normalize a JSONL field name to a Prometheus-safe metric name.
* Per Prometheus 3.0 rules: [a-zA-Z_:][a-zA-Z0-9_:]*. Replace dot/dash/space → '_'.
*/
function normalizeMetricName(name) {
const safe = String(name).toLowerCase().replace(/[.\-\s]+/g, '_').replace(/[^a-zA-Z0-9_:]/g, '_');
return METRIC_PREFIX + safe;
}
/**
* Quote a Prometheus label value per spec § Format: backslash, double-quote, newline.
*/
function escapeLabel(v) {
return String(v).replace(/\\/g, '\\\\').replace(/"/g, '\\"').replace(/\n/g, '\\n');
}
/**
* Determine which fields in a record are numeric metrics vs string labels.
* Numeric → emitted as samples. String/bool → labels on the sample.
*/
function partitionRecord(record) {
const labels = {};
const metrics = {};
for (const [k, v] of Object.entries(record)) {
if (k === 'ts') continue; // never emit ts as label nor metric (Prom server timestamps)
if (typeof v === 'number') metrics[k] = v;
else if (typeof v === 'boolean') metrics[k] = v ? 1 : 0;
else if (typeof v === 'string') labels[k] = v;
// Skip arrays/objects — caller's allowlist should have flattened these.
}
return { labels, metrics };
}
/**
* Group records by their schema_id (an extracted convention: schema_id is the
* caller-provided identifier; if absent, fall back to 'unknown').
*/
function groupBySchema(records) {
const groups = new Map();
for (const r of records) {
const id = (r && typeof r._schema_id === 'string') ? r._schema_id : 'unknown';
if (!groups.has(id)) groups.set(id, []);
groups.get(id).push(r);
}
return groups;
}
/**
* Transform JSONL records into Prometheus text-format. Pure function.
*
* @param {Array<object>} records Allowlist-redacted records (caller responsibility).
* @param {{help?: object}} [opts] Optional: help-text overrides per metric (object).
* @returns {string} Prometheus text-format. Empty input → empty string.
*/
export function transformToPrometheus(records, opts = {}) {
if (!Array.isArray(records) || records.length === 0) return '';
const lines = [];
const helpMap = opts.help || {};
const groups = groupBySchema(records);
// Track which metric names we've emitted HELP/TYPE for (per Prometheus spec:
// emit HELP/TYPE once per metric, then all samples for that metric).
const emittedMeta = new Set();
// First pass: collect all unique (schema, metric_field) → sample lines
// grouped by metric so HELP/TYPE come before all samples for that metric.
const samplesByMetric = new Map();
for (const [schemaId, group] of groups.entries()) {
for (const record of group) {
const { labels, metrics } = partitionRecord(record);
const labelStr = Object.entries(labels)
.map(([k, v]) => `${k}="${escapeLabel(v)}"`)
.join(',');
const labelBlock = labelStr ? `{${labelStr}}` : '';
for (const [metricField, value] of Object.entries(metrics)) {
const metricName = normalizeMetricName(`${schemaId}_${metricField}`);
if (!samplesByMetric.has(metricName)) {
samplesByMetric.set(metricName, []);
}
samplesByMetric.get(metricName).push(`${metricName}${labelBlock} ${value}`);
}
}
}
// Sort metric names for deterministic output (snapshot-test-friendly)
const sortedMetrics = [...samplesByMetric.keys()].sort();
for (const metricName of sortedMetrics) {
const help = helpMap[metricName] || `voyage stats — ${metricName.slice(METRIC_PREFIX.length)}`;
const type = inferMetricType(metricName);
if (!emittedMeta.has(metricName)) {
lines.push(`# HELP ${metricName} ${help}`);
lines.push(`# TYPE ${metricName} ${type}`);
emittedMeta.add(metricName);
}
// Sort samples for determinism
const samples = samplesByMetric.get(metricName).sort();
for (const s of samples) lines.push(s);
}
return lines.join('\n') + (lines.length > 0 ? '\n' : '');
}
/**
* Heuristic: counter for *_total / *_count / *_passed / *_failed; histogram for
* *_ms / *_duration / *_p50 / *_p99; gauge for everything else (per Prometheus
* conventions). Pure & deterministic.
*/
function inferMetricType(metricName) {
if (/_total$|_count$|_passed$|_failed$|_skipped$/.test(metricName)) return 'counter';
if (/_ms$|_duration|_p\d+$|_seconds$/.test(metricName)) return 'histogram';
return 'gauge';
}
export { normalizeMetricName, partitionRecord, inferMetricType };

89
lib/review/gold-scorer.mjs
View file

@ -1,89 +0,0 @@
// lib/review/gold-scorer.mjs
// SKAL-1·4b — offline gold-scored output eval scorer.
//
// Scores a recorded agent-run's findings against a golden corpus record
// (voyage-eval-gold/1, see docs/eval-corpus/README.md) at (file, rule_key)
// granularity. line and severity are deliberately NOT part of the match key:
// the eval asks "did the run surface this issue at all", and (file, rule_key)
// is the identity pair a delivered diff + the catalogue uniquely determine.
//
// Pairs with coordinator-contract.mjs (4a): that module turns committed
// reviewer payloads into the run findings scored here. This module is the
// offline counterpart to the live LLM-in-the-loop eval (4c) — pure: no I/O,
// no LLM, no network, no time, no randomness.
//
// Vacuous-set conventions (documented so the degenerate numbers are honest):
// - precision when nothing is predicted (tp+fp === 0) -> 1 (no false positives)
// - recall when there is nothing to find (tp+fn === 0) -> 1 (found all zero)
// - f1 collapses to 0 whenever precision or recall is 0, so it stays the
// honest single-number summary for the degenerate run/gold cases.
// NUL separator: file paths and rule_keys never contain it, so the join is
// an injective (file, rule_key) -> string key.
const SEP = '\x00';
function pairKey(f) {
return `${f.file}${SEP}${f.rule_key}`;
}
// Human-readable form of a pair key (for matched/missed/spurious reporting).
function pairLabel(key) {
return key.replace(SEP, ' ');
}
// Set of unique (file, rule_key) pair keys from a findings list. Findings
// missing either field are skipped (they cannot identify a pair).
function pairSet(findings) {
const s = new Set();
for (const f of findings ?? []) {
if (f && typeof f.file === 'string' && typeof f.rule_key === 'string') {
s.add(pairKey(f));
}
}
return s;
}
/**
* Score run findings against gold findings at (file, rule_key) granularity.
* @param {object[]|null|undefined} runFindings findings produced by the recorded run
* @param {object[]|null|undefined} goldFindings the golden corpus findings
* @returns {{
* tp: number, fp: number, fn: number,
* precision: number, recall: number, f1: number,
* matched: string[], missed: string[], spurious: string[]
* }} matched/missed/spurious are "<file> <rule_key>" labels.
*/
export function scoreFindings(runFindings, goldFindings) {
const runPairs = pairSet(runFindings);
const goldPairs = pairSet(goldFindings);
const matched = [];
const spurious = [];
for (const p of runPairs) {
(goldPairs.has(p) ? matched : spurious).push(pairLabel(p));
}
const missed = [];
for (const p of goldPairs) {
if (!runPairs.has(p)) missed.push(pairLabel(p));
}
const tp = matched.length;
const fp = spurious.length;
const fn = missed.length;
const precision = tp + fp === 0 ? 1 : tp / (tp + fp);
const recall = tp + fn === 0 ? 1 : tp / (tp + fn);
const f1 = precision + recall === 0 ? 0 : (2 * precision * recall) / (precision + recall);
return { tp, fp, fn, precision, recall, f1, matched, missed, spurious };
}
/**
* Exact match of a run's coordinator verdict against the gold expected_verdict.
* @param {string} runVerdict
* @param {string} goldVerdict
* @returns {boolean}
*/
export function scoreVerdict(runVerdict, goldVerdict) {
return runVerdict === goldVerdict;
}

142
lib/stats/cache-analyzer.mjs
View file

@ -1,142 +0,0 @@
// lib/stats/cache-analyzer.mjs
// Summarizes trekexecute-stats.jsonl: total events, percentile wall times,
// time range. Companion to event-emit.mjs (which produces the jsonl).
//
// Designed for /trekplan Spor C: gives C3 telemetry context when
// interpreting Q3 experiment numbers (5+ weeks of accumulated data on the
// operator's machine as of 2026-05-04).
//
// Zero npm dependencies. Node stdlib only.
import { readFileSync, existsSync } from 'node:fs';
import { isMain } from '../util/is-main.mjs';
function usage() {
return `cache-analyzer.mjs — summarize trekexecute-stats.jsonl
USAGE:
node lib/stats/cache-analyzer.mjs --json <path-to-jsonl>
OUTPUT (stdout, JSON):
{
"total_events": <n>,
"events_with_duration": <n>,
"wall_time_ms_p50": <ms or null>,
"wall_time_ms_p90": <ms or null>,
"wall_time_ms_max": <ms or null>,
"unique_event_names": [...],
"oldest_event_iso": "<iso8601 or null>",
"newest_event_iso": "<iso8601 or null>"
}
EXIT:
0 success, 1 file not found / read error, 2 usage error.
`;
}
export function summarize(lines) {
const summary = {
total_events: 0,
events_with_duration: 0,
wall_time_ms_p50: null,
wall_time_ms_p90: null,
wall_time_ms_max: null,
unique_event_names: [],
oldest_event_iso: null,
newest_event_iso: null,
// SKAL-2 token/cost aggregation (additive; zero when no token records).
// With upsert semantics (one record per session_id), summing across lines
// = correct cross-session aggregate. cost_usd is summed only when finite;
// a null (refuse-to-estimate) record still counts in sessions_with_tokens.
total_tokens_input: 0,
total_tokens_output: 0,
total_tokens_cache_creation: 0,
total_tokens_cache_read: 0,
total_cost_usd: 0,
sessions_with_tokens: 0,
};
const durations = [];
const names = new Set();
let oldestMs = null;
let newestMs = null;
for (const line of lines) {
const trimmed = line.trim();
if (trimmed === '') continue;
let obj;
try { obj = JSON.parse(trimmed); }
catch { continue; }
summary.total_events++;
if (obj.event && typeof obj.event === 'string') names.add(obj.event);
else if (obj.name && typeof obj.name === 'string') names.add(obj.name);
if (typeof obj.duration_ms === 'number' && Number.isFinite(obj.duration_ms)) {
durations.push(obj.duration_ms);
summary.events_with_duration++;
}
const tsField = obj.timestamp || obj.ts || obj.iso || obj.time;
if (typeof tsField === 'string') {
const t = Date.parse(tsField);
if (!Number.isNaN(t)) {
if (oldestMs === null || t < oldestMs) oldestMs = t;
if (newestMs === null || t > newestMs) newestMs = t;
}
}
// SKAL-2: aggregate token/cost from token-bearing records (token-usage
// schema). Detect by presence of any numeric token field.
const tokenKeys = ['tokens_input', 'tokens_output', 'tokens_cache_creation', 'tokens_cache_read'];
const hasTokens = tokenKeys.some(k => typeof obj[k] === 'number' && Number.isFinite(obj[k]));
if (hasTokens) {
summary.sessions_with_tokens++;
if (Number.isFinite(obj.tokens_input)) summary.total_tokens_input += obj.tokens_input;
if (Number.isFinite(obj.tokens_output)) summary.total_tokens_output += obj.tokens_output;
if (Number.isFinite(obj.tokens_cache_creation)) summary.total_tokens_cache_creation += obj.tokens_cache_creation;
if (Number.isFinite(obj.tokens_cache_read)) summary.total_tokens_cache_read += obj.tokens_cache_read;
// cost_usd may be null (refuse-to-estimate) — sum finite values only.
if (Number.isFinite(obj.cost_usd)) summary.total_cost_usd += obj.cost_usd;
}
}
if (durations.length > 0) {
durations.sort((a, b) => a - b);
const p50Idx = Math.floor(durations.length * 0.5);
const p90Idx = Math.floor(durations.length * 0.9);
summary.wall_time_ms_p50 = durations[Math.min(p50Idx, durations.length - 1)];
summary.wall_time_ms_p90 = durations[Math.min(p90Idx, durations.length - 1)];
summary.wall_time_ms_max = durations[durations.length - 1];
}
summary.unique_event_names = [...names].sort();
if (oldestMs !== null) summary.oldest_event_iso = new Date(oldestMs).toISOString();
if (newestMs !== null) summary.newest_event_iso = new Date(newestMs).toISOString();
return summary;
}
export function summarizeFile(path) {
if (!existsSync(path)) {
return { error: `file not found: ${path}` };
}
let text;
try { text = readFileSync(path, 'utf-8'); }
catch (e) { return { error: `read error: ${e.message}` }; }
return summarize(text.split('\n'));
}
if (isMain(import.meta.url)) {
const args = process.argv.slice(2);
const jsonIdx = args.indexOf('--json');
if (jsonIdx === -1 || !args[jsonIdx + 1]) {
process.stderr.write(usage());
process.exit(2);
}
const path = args[jsonIdx + 1];
const result = summarizeFile(path);
if (result.error) {
process.stderr.write(`cache-analyzer: ${result.error}\n`);
process.exit(1);
}
process.stdout.write(JSON.stringify(result, null, 2) + '\n');
process.exit(0);
}

126
lib/stats/claimed-order.mjs
View file

@ -1,126 +0,0 @@
// lib/stats/claimed-order.mjs
// Which coord-queue order is this Voyage run working on? Read from the queue
// itself, never typed by the model and never guessed.
//
// Why this exists: the yardstick's `use` number joins a new-work order to a
// Voyage run. A key declared on the order before any run (a `slug:`) is not
// proof a run happened, and no order carried one. The run itself has to say
// which order it ran, so /trekbrief's stats record carries the id this module
// finds (lib/stats/stats-append.mjs adds it; the model never copies it).
//
// The contract it mirrors is repo-mailbox's claim verb (coord-order-claim.sh,
// 0.37.0), read here, never called:
// - coord root ${CLAUDE_COORD_DIR:-$HOME/.claude/coord}
// - mailbox basename of `git rev-parse --show-toplevel`; `_*` is reserved
// - claim `mv orders/<id>.md orders/claimed/<id>.md`, so the id is the
// file stem, plus a best-effort sidecar `claimed/<id>.claim`
// holding `claimed-at: <ISO>`
// - close coord-order-done.sh deletes the `.claim`, so the claim time
// survives only in the record this module feeds.
// A worktree's toplevel basename is the worktree's name, so the main repo's
// name (the parent of the common git dir) is searched too.
//
// Exactly one claimed order -> { order_id, order_claimed_at, order_reason: null }.
// Anything else -> order_id null (or the id with `no-claim-time`) and a reason:
// mailbox-unresolved · coord-missing · coord-unreadable · none-claimed ·
// several-claimed:N · to-mismatch · no-claim-time. It only reads, never throws,
// and starts nothing. Zero npm dependencies.
import { existsSync, readdirSync, readFileSync, realpathSync, statSync } from 'node:fs';
import { homedir } from 'node:os';
import { basename, dirname, isAbsolute, join, relative, resolve } from 'node:path';
import { execFileSync } from 'node:child_process';
const GIT_TIMEOUT_MS = 5000;
const unattributed = (order_reason) => ({ order_id: null, order_claimed_at: null, order_reason });
/** The coord root the claim script uses: CLAUDE_COORD_DIR, else $HOME/.claude/coord. */
export function resolveCoordDir(env = process.env) {
if (env.CLAUDE_COORD_DIR) return env.CLAUDE_COORD_DIR;
const home = env.HOME && env.HOME.length > 0 ? env.HOME : homedir();
return join(home, '.claude', 'coord');
}
function gitLine(dir, args, env) {
try {
return execFileSync('git', ['-C', dir, 'rev-parse', ...args], {
encoding: 'utf8', timeout: GIT_TIMEOUT_MS, env, stdio: ['ignore', 'pipe', 'ignore'],
}).trim();
} catch {
return '';
}
}
/**
* The mailbox names `dir` can be filed under: the toplevel's basename (the
* claim script's rule) and, in a linked worktree, the main repo's name. Two
* independent git calls, so one failing never loses the other. An inherited
* GIT_DIR / GIT_WORK_TREE would point git at another repo, so both are dropped.
*/
export function resolveMailboxes(dir, { env = {} } = {}) {
const childEnv = { ...process.env, ...env };
for (const key of ['GIT_DIR', 'GIT_WORK_TREE', 'GIT_COMMON_DIR', 'GIT_INDEX_FILE']) delete childEnv[key];
const names = [];
const top = gitLine(dir, ['--show-toplevel'], childEnv);
if (top) names.push(basename(top));
const common = gitLine(dir, ['--path-format=absolute', '--git-common-dir'], childEnv);
// Only `<repo>/.git` names a repo; a submodule's `.git/modules/<x>` does not.
if (common && basename(common) === '.git') names.push(basename(dirname(common)));
return [...new Set(names)];
}
const isDir = (p) => existsSync(p) && statSync(p).isDirectory();
/** The one claimed order across `mailboxes`, or null and the reason it is not one. */
export function findClaimedOrder({ coordDir, mailboxes }) {
try {
if (!isDir(coordDir)) return unattributed('coord-missing');
const names = (mailboxes ?? []).filter((m) => typeof m === 'string' && m !== '' && !m.startsWith('_'));
if (names.length === 0) return unattributed('mailbox-unresolved');
const found = [];
for (const mailbox of names) {
const dir = join(coordDir, mailbox, 'orders', 'claimed');
if (!isDir(dir)) continue;
for (const name of readdirSync(dir).sort()) {
if (name.endsWith('.md')) found.push({ mailbox, dir, id: name.slice(0, -3) });
}
}
if (found.length === 0) return unattributed('none-claimed');
if (found.length > 1) return unattributed(`several-claimed:${found.length}`);
const [{ mailbox, dir, id }] = found;
const text = readFileSync(join(dir, `${id}.md`), 'utf8');
const fm = text.match(/^---\r?\n([\s\S]*?)\r?\n---/)?.[1] ?? '';
const to = fm.match(/^to:[ \t]*(.*?)[ \t]*$/m)?.[1];
if (to !== undefined && to !== mailbox) return unattributed('to-mismatch');
const claimPath = join(dir, `${id}.claim`);
const at = existsSync(claimPath)
? readFileSync(claimPath, 'utf8').match(/^claimed-at:[ \t]*(\S+)[ \t]*$/m)?.[1]
: undefined;
if (!at) return { order_id: id, order_claimed_at: null, order_reason: 'no-claim-time' };
return { order_id: id, order_claimed_at: at, order_reason: null };
} catch {
return unattributed('coord-unreadable');
}
}
// The directory to resolve the mailbox from: `projectDir` when it is a directory
// inside `cwd` (a brief's project dir), else `cwd`. A path that walks out of cwd
// cannot move the lookup to another repo.
function baseDir(cwd, projectDir) {
if (typeof projectDir !== 'string' || projectDir === '') return cwd;
try {
const abs = resolve(cwd, projectDir);
if (!isDir(abs)) return cwd;
const rel = relative(realpathSync(cwd), realpathSync(abs));
return rel.startsWith('..') || isAbsolute(rel) ? cwd : abs;
} catch {
return cwd;
}
}
/** The attribution a /trekbrief record carries: always the three keys. */
export function attributeOrder({ cwd = process.cwd(), env = process.env, projectDir } = {}) {
const mailboxes = resolveMailboxes(baseDir(cwd, projectDir), { env });
if (mailboxes.length === 0) return unattributed('mailbox-unresolved');
return findClaimedOrder({ coordDir: resolveCoordDir(env), mailboxes });
}

129
lib/stats/event-emit.mjs
View file

@ -1,129 +0,0 @@
// lib/stats/event-emit.mjs
// Atomic JSONL append for autonomy-lifecycle events (plan-v2 Step 6).
//
// Writes one line per event to <data dir>/trekexecute-stats.jsonl, where the data
// dir is resolved by stats-append.mjs's resolveStatsDir: CLAUDE_PLUGIN_DATA when
// the harness provides it, else the plugin's directory under ~/.claude/plugins/data/,
// the one scripts/yardstick.mjs reads (one resolver, no second copy here). The Bash tool env carries no
// CLAUDE_PLUGIN_DATA, so the old "unset -> silent skip" dropped every event the
// command prose emitted. A failed write is REPORTED ({written:false, reason}),
// never thrown: stats still never block the workflow.
//
// Every emission carries:
// - ts : ISO-8601 timestamp (REQUIRED per SC4 contract)
// - event : the requested event name
// - known_event : true for recognized events, false otherwise
// - payload : caller-supplied object (may be {})
//
// Recognized events: brief-approved, main-merge-gate, main-merge-approved,
// main-merge-declined, gates_mode_flag_override, user_input. Unknown event names
// are still emitted (with known_event: false) so that the audit trail is
// complete; downstream consumers filter as needed.
//
// CLI shim:
// node lib/stats/event-emit.mjs --event brief-approved --payload '{...}'
// node lib/stats/event-emit.mjs --event main-merge-gate --payload - <<'JSON' (payload on stdin)
// → prints {"written":true,"path":...} exit 0 | {"written":false,"reason":...} exit 1
// (usage error, payload-not-json and a failed write alike). The command prose
// prints the reason and continues.
//
// Import direction: event-emit imports stats-append; stats-append must never
// import event-emit (that would be a cycle).
import { appendFileSync, existsSync, mkdirSync, readFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { resolveStatsDir } from './stats-append.mjs';
import { isMain } from '../util/is-main.mjs';
export const KNOWN_EVENTS = Object.freeze(new Set([
'brief-approved',
'main-merge-gate',
'main-merge-approved',
'main-merge-declined',
'gates_mode_flag_override',
'user_input',
]));
const STATS_FILENAME = 'trekexecute-stats.jsonl';
/**
* Resolve the stats file path: CLAUDE_PLUGIN_DATA when set, else the directory
* stats-append.mjs and scripts/yardstick.mjs use. Never null.
*/
export function resolveStatsPath(env = process.env) {
return join(resolveStatsDir(env), STATS_FILENAME);
}
/**
* Build the JSON record. Pure — no I/O.
*/
export function buildRecord(event, payload = {}, now = new Date()) {
if (typeof event !== 'string' || event.length === 0) {
throw new TypeError('event must be a non-empty string');
}
return {
ts: now.toISOString(),
event,
known_event: KNOWN_EVENTS.has(event),
payload: (payload && typeof payload === 'object') ? payload : {},
};
}
/**
* Emit an event. Never throws — a failed write is returned as {written:false,
* reason} because lifecycle telemetry must not block the user's workflow.
*
* @returns {{ written: boolean, path: string | null, reason?: string }}
*/
export function emit(event, payload = {}, opts = {}) {
const env = opts.env || process.env;
const now = opts.now || new Date();
let record;
try {
record = buildRecord(event, payload, now);
} catch (e) {
return { written: false, path: null, reason: `record-build: ${e.message}` };
}
const path = opts.path || resolveStatsPath(env);
try {
const dir = dirname(path);
if (!existsSync(dir)) {
// A dir that cannot be created is reported, never thrown.
try { mkdirSync(dir, { recursive: true }); } catch { return { written: false, path, reason: 'dir-mkdir-failed' }; }
}
appendFileSync(path, JSON.stringify(record) + '\n');
return { written: true, path };
} catch (e) {
return { written: false, path, reason: `append-failed: ${e.message}` };
}
}
// ---- CLI shim ----------------------------------------------------------------
function parseArgs(argv) {
const out = {};
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === '--event') out.event = argv[++i];
else if (a === '--payload') out.payload = argv[++i];
}
return out;
}
if (isMain(import.meta.url)) {
const args = parseArgs(process.argv.slice(2));
let result;
if (!args.event) {
result = { written: false, path: null, reason: 'usage: --event NAME [--payload JSON]' };
} else {
let payload = {};
let bad = false;
if (args.payload) {
// `-` = read it from stdin: a quoted heredoc passes it with nothing expanded.
try { payload = JSON.parse(args.payload === '-' ? readFileSync(0, 'utf8') : args.payload); } catch { bad = true; }
}
result = bad ? { written: false, path: null, reason: 'payload-not-json' } : emit(args.event, payload);
}
process.stdout.write(JSON.stringify(result) + '\n');
process.exit(result.written ? 0 : 1);
}

83
lib/stats/stats-append.mjs
View file

@ -1,83 +0,0 @@
// lib/stats/stats-append.mjs
// Append one command-stats record to <data dir>/<kind>-stats.jsonl.
//
// Why this exists: the commands' stats phases said "append to
// ${CLAUDE_PLUGIN_DATA}/<kind>-stats.jsonl; if it is not set, skip silently".
// The Bash tool's env carries no CLAUDE_PLUGIN_DATA, so the record was skipped
// whenever the model did not guess the directory itself — the same silent skip
// that left brief-approved at 0 records. This writer resolves the directory
// the way scripts/yardstick.mjs reads it, and a failed write is REPORTED
// (exit 1 + reason), never swallowed. Stats still never block the workflow:
// the command prose reports the failure and carries on.
//
// A `trekbrief` record also carries the coord-queue order the run is working
// on: order_id, order_claimed_at, order_reason, found by
// lib/stats/claimed-order.mjs in the queue itself. The WRITER adds them - any
// order_* key the record arrived with is dropped first - so the model never
// types an order id, and the keys are always present (null + reason when no
// single order can be named; absent keys mean the record predates this).
//
// CLI (the record is read from stdin, so no shell quoting of JSON):
// node lib/stats/stats-append.mjs <kind> <<'JSON'
// { ...record... }
// JSON
// → prints {"written":true,"path":...} exit 0 | {"written":false,"reason":...} exit 1
import { appendFileSync, mkdirSync, readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { join } from 'node:path';
import { attributeOrder } from './claimed-order.mjs';
import { isMain } from '../util/is-main.mjs';
export const STATS_KINDS = Object.freeze([
'trekbrief', 'trekresearch', 'trekplan', 'trekexecute', 'trekreview', 'trekcontinue',
]);
/**
* The plugin data directory: CLAUDE_PLUGIN_DATA when the harness provides it,
* otherwise the directory the hooks write to and scripts/yardstick.mjs reads.
*/
export function resolveStatsDir(env = process.env) {
if (env.CLAUDE_PLUGIN_DATA) return env.CLAUDE_PLUGIN_DATA;
const home = env.HOME && env.HOME.length > 0 ? env.HOME : homedir();
return join(home, '.claude', 'plugins', 'data', 'voyage-ktg-plugin-marketplace');
}
/**
* @returns {{ written: boolean, path: string|null, reason?: string }}
*/
export function appendStatsRecord(kind, record, env = process.env, { cwd = process.cwd() } = {}) {
if (!STATS_KINDS.includes(kind)) {
return { written: false, path: null, reason: `unknown kind "${kind}" (one of ${STATS_KINDS.join(', ')})` };
}
if (!record || typeof record !== 'object' || Array.isArray(record)) {
return { written: false, path: null, reason: 'record must be a JSON object' };
}
if (kind === 'trekbrief') {
const own = Object.fromEntries(Object.entries(record).filter(([k]) => !k.startsWith('order_')));
record = { ...own, ...attributeOrder({ cwd, env, projectDir: record.project_dir }) };
}
const dir = resolveStatsDir(env);
const path = join(dir, `${kind}-stats.jsonl`);
try {
mkdirSync(dir, { recursive: true });
appendFileSync(path, JSON.stringify(record) + '\n');
} catch (e) {
return { written: false, path, reason: `${e.code || 'write-failed'}: ${e.message}` };
}
return { written: true, path };
}
if (isMain(import.meta.url)) {
const kind = process.argv[2];
let record = null;
let reason = null;
try {
record = JSON.parse(readFileSync(0, 'utf8'));
} catch (e) {
reason = `stdin is not JSON: ${e.message}`;
}
const r = reason ? { written: false, path: null, reason } : appendStatsRecord(kind, record);
process.stdout.write(JSON.stringify(r) + '\n');
process.exit(r.written ? 0 : 1);
}

234
lib/stats/token-usage.mjs
View file

@ -1,234 +0,0 @@
// lib/stats/token-usage.mjs
// SKAL-2 — pure token-usage parser + cache-aware cost derivation for Voyage
// observability. Captures MAIN-CONTEXT token/cost from the Claude Code
// transcript (transcript_path). Each assistant record carries message.usage
// with a per-REQUEST snapshot.
//
// Verified 2026-06-26 against a real local transcript (216 lines): 70
// assistant records collapse to 31 distinct requestIds — records duplicate
// per requestId (streamed snapshots), so we dedup by requestId keeping the
// LAST occurrence (GH #28197 streaming-placeholder mitigation), then sum
// across requests. The main transcript carried zero isSidechain:true records.
//
// v1 scope = MAIN-CONTEXT only. Sub-agent (swarm) turns live in separate
// agent-*.jsonl siblings and are a documented v2 follow-on — so the
// main-transcript sum UNDER-counts total Voyage cost. Every record is stamped
// scope:'main-context' so no reader mistakes cost_usd for the session total.
//
// The four core functions (parseTranscriptUsage, deriveCost, buildRecord,
// upsertSessionRecord) are PURE (no I/O). captureTokenUsage is the impure
// shell that wires them to the filesystem (read transcript → upsert jsonl).
//
// Zero npm dependencies. Node stdlib only.
import { readFileSync, existsSync, writeFileSync, renameSync, statSync } from 'node:fs';
import { join, dirname } from 'node:path';
// Per-Mtok USD prices, resolved 2026-06-26 via the claude-api skill reference:
// base input/output from the model table; cache rates from the prompt-caching
// doc multipliers (cache_read 0.1x, write_5m 1.25x, write_1h 2.0x of input).
// claude-fable-5 resolved 2026-07-02 from the official platform pricing docs.
export const PRICE_TABLE = Object.freeze({
'claude-opus-4-8': Object.freeze({
input: 5.0,
output: 25.0,
cache_read: 0.5,
cache_write_5m: 6.25,
cache_write_1h: 10.0,
}),
'claude-fable-5': Object.freeze({
input: 10.0,
output: 50.0,
cache_read: 1.0,
cache_write_5m: 12.5,
cache_write_1h: 20.0,
}),
});
// Date the PRICE_TABLE values were resolved/verified. Bump when prices change.
export const PRICE_TABLE_VERSION = '2026-07-02';
function num(v) {
return typeof v === 'number' && Number.isFinite(v) ? v : 0;
}
/**
* Single pass over transcript JSONL text. Keeps only main-chain assistant
* records (type==='assistant', isSidechain!==true), dedups by requestId
* keeping the LAST occurrence (per-request usage snapshots duplicate across
* streamed records), and sums the four usage token fields. Malformed lines
* are skipped.
*
* @returns {{tokens_input:number, tokens_output:number,
* tokens_cache_creation:number, tokens_cache_read:number}}
*/
export function parseTranscriptUsage(text) {
const lines = (text || '').split('\n');
// requestId -> usage (last occurrence wins). Records without a requestId
// get a unique sentinel key so each is still counted exactly once.
const byRequest = new Map();
let anon = 0;
for (const line of lines) {
const trimmed = line.trim();
if (trimmed === '') continue;
let obj;
try { obj = JSON.parse(trimmed); }
catch { continue; }
if (!obj || obj.type !== 'assistant') continue;
if (obj.isSidechain === true) continue;
const usage = obj.message && obj.message.usage;
if (!usage || typeof usage !== 'object') continue;
const key = typeof obj.requestId === 'string' && obj.requestId
? obj.requestId
: `__anon_${anon++}`;
byRequest.set(key, usage); // last occurrence wins
}
const totals = {
tokens_input: 0,
tokens_output: 0,
tokens_cache_creation: 0,
tokens_cache_read: 0,
};
for (const usage of byRequest.values()) {
totals.tokens_input += num(usage.input_tokens);
totals.tokens_output += num(usage.output_tokens);
totals.tokens_cache_creation += num(usage.cache_creation_input_tokens);
totals.tokens_cache_read += num(usage.cache_read_input_tokens);
}
return totals;
}
/**
* Cache-aware USD cost. Lumped cache_creation is priced at the 5m write rate
* (Claude Code's default cache TTL; verified the dominant case on a real
* transcript — the 1h split is a documented v2 refinement). When the model is
* absent from the table we REFUSE to estimate: {cost_usd:null, is_estimate:true}.
*
* @returns {{cost_usd:number|null, is_estimate:boolean}}
*/
export function deriveCost(totals, model, priceTable = PRICE_TABLE) {
const price = priceTable && priceTable[model];
if (!price) return { cost_usd: null, is_estimate: true };
const t = totals || {};
const cost =
(num(t.tokens_input) * price.input +
num(t.tokens_output) * price.output +
num(t.tokens_cache_creation) * price.cache_write_5m +
num(t.tokens_cache_read) * price.cache_read) / 1_000_000;
return { cost_usd: cost, is_estimate: false };
}
/**
* Build the flat numeric record. Stamps scope:'main-context' and
* price_table_version. session_id is written for upsert keying but MUST be
* stripped at export (CWE-212, enforced by the field allowlist in Step 2).
*
* @param {{sessionId:string, model:string, totals:object,
* priceTable?:object, now:string}} args
*/
export function buildRecord({ sessionId, model, totals, priceTable = PRICE_TABLE, now }) {
const t = totals || {};
const { cost_usd, is_estimate } = deriveCost(t, model, priceTable);
return {
ts: now,
session_id: sessionId,
scope: 'main-context',
model,
tokens_input: num(t.tokens_input),
tokens_output: num(t.tokens_output),
tokens_cache_creation: num(t.tokens_cache_creation),
tokens_cache_read: num(t.tokens_cache_read),
cost_usd,
is_estimate,
price_table_version: PRICE_TABLE_VERSION,
};
}
/**
* Pure read-modify-write. Parses existing JSONL lines, REPLACES the line whose
* session_id matches the record (dropping any further same-session dupes),
* else APPENDS. Returns the new file text (one line per session, trailing
* newline). One-line-per-session is what keeps cross-session sums from
* double-counting (vs append-and-grow).
*/
export function upsertSessionRecord(existingText, record) {
const lines = (existingText || '').split('\n');
const out = [];
let replaced = false;
for (const line of lines) {
const trimmed = line.trim();
if (trimmed === '') continue;
let obj;
try { obj = JSON.parse(trimmed); }
catch { out.push(trimmed); continue; } // preserve unparseable lines verbatim
if (obj && obj.session_id === record.session_id) {
if (!replaced) { out.push(JSON.stringify(record)); replaced = true; }
// else: duplicate same-session line — drop it
} else {
out.push(trimmed);
}
}
if (!replaced) out.push(JSON.stringify(record));
return out.join('\n') + '\n';
}
/**
* Last main-chain (non-sidechain) assistant model in the transcript. Used to
* pick the price-table key. Returns null when no model is found (→ deriveCost
* refuses to estimate). Pure.
*/
export function lastMainChainModel(text) {
const lines = (text || '').split('\n');
let model = null;
for (const line of lines) {
const trimmed = line.trim();
if (trimmed === '') continue;
let obj;
try { obj = JSON.parse(trimmed); }
catch { continue; }
if (!obj || obj.type !== 'assistant' || obj.isSidechain === true) continue;
const m = obj.message && obj.message.model;
if (typeof m === 'string' && m) model = m;
}
return model;
}
/**
* Impure capture shell (Step 4). Reads the transcript, derives main-context
* token totals + cost, and UPSERTS a one-line-per-session record into
* {dataDir}/token-usage-stats.jsonl via an atomic temp+rename write
* (EXDEV mitigation: tmp lives in the same dir as the target).
*
* Returns the written record, or null when skipped (no path / no dataDir /
* transcript not a readable regular file). Throws propagate to the caller —
* the Stop hook wraps this in try/catch so capture stays fail-open.
*
* @param {{transcriptPath:string, sessionId:string, dataDir:string,
* now?:string}} args
*/
export function captureTokenUsage({ transcriptPath, sessionId, dataDir, now }) {
if (!transcriptPath || !dataDir) return null;
let st;
try { st = statSync(transcriptPath); }
catch { return null; }
if (!st.isFile()) return null;
const text = readFileSync(transcriptPath, 'utf-8');
const totals = parseTranscriptUsage(text);
const model = lastMainChainModel(text);
const record = buildRecord({
sessionId,
model,
totals,
now: now || new Date().toISOString(),
});
const outPath = join(dataDir, 'token-usage-stats.jsonl');
const existing = existsSync(outPath) ? readFileSync(outPath, 'utf-8') : '';
const updated = upsertSessionRecord(existing, record);
const tmpPath = join(dirname(outPath), '.token-usage-stats.jsonl.tmp');
writeFileSync(tmpPath, updated);
renameSync(tmpPath, outPath);
return record;
}

2
lib/util/research-loop-cap.mjs
View file

@ -27,7 +27,7 @@
// cannot be WRITTEN denies the turn, and a ledger that exists but cannot be
// READ denies it too. Only ENOENT counts as zero turns spent, because that is
// the legitimate first-turn state. This module is a budget control, not
// telemetry — the opposite of lib/stats/event-emit.mjs's fail-open.
// telemetry.
//
// CLI shim, two modes:
// node lib/util/research-loop-cap.mjs --run-id ID --dimension D --effort E

23
lib/validators/intent-approval.mjs
View file

@ -34,7 +34,7 @@
// node lib/validators/intent-approval.mjs --check [--json] <brief.md>
// exit 0 = approved and current; exit 1 = stop (errors); exit 2 = usage
// node lib/validators/intent-approval.mjs --stamp <brief.md>
// writes the marker and emits `brief-approved`; exit 0 stamped, 1 refused
// writes the marker; exit 0 stamped, 1 refused
// node lib/validators/intent-approval.mjs --resolve <project-dir | brief-file>
// what `/trekbrief --approve` approves: {brief_path, project_dir, plan_command}; exit 1 = no brief
@ -43,15 +43,12 @@ import { existsSync, readFileSync, renameSync, statSync, writeFileSync } from 'n
import { basename, dirname, join } from 'node:path';
import { parseDocument } from '../util/frontmatter.mjs';
import { issue } from '../util/result.mjs';
import { emit } from '../stats/event-emit.mjs';
import { resolveStatsDir } from '../stats/stats-append.mjs';
import { isMain } from '../util/is-main.mjs';
export const INTENT_HASH_FIELD = 'intent_approved_hash';
export const INTENT_APPROVED_AT_FIELD = 'intent_approved_at';
export const INTENT_SECTIONS = Object.freeze(['Intent', 'Goal']);
const HASH_RE = /^sha256:[0-9a-f]{64}$/;
const STATS_FILENAME = 'trekexecute-stats.jsonl'; // where event-emit writes lifecycle events
const APPROVE_HINT = 'Run /trekbrief --approve <project-dir | brief-file> — the operator reads ## Intent + ## Goal and answers "Approve".';
@ -200,18 +197,6 @@ export function stampIntentApproval(text, now = new Date()) {
return { stamped: true, text: out, hash, approvedAt };
}
/**
* Where the brief-approved record goes. CLAUDE_PLUGIN_DATA when the harness
* provides it; otherwise the plugin data dir the hooks write to and
* scripts/yardstick.mjs reads — the Bash tool env carries no CLAUDE_PLUGIN_DATA.
* event-emit's fallback used to be a silent skip, which is how brief-approved
* reached 0 records; it now resolves the same directory. One resolver for every
* stats writer: lib/stats/stats-append.mjs.
*/
export function resolveApprovalDataDir(env = process.env) {
return resolveStatsDir(env);
}
/**
* What `/trekbrief --approve <arg>` approves. A directory means `<dir>/brief.md`
* (a /trekplan --project brief); a file is itself (a /trekplan --brief brief,
@ -278,12 +263,8 @@ if (isMain(import.meta.url)) {
const tmp = join(dirname(file), `.${basename(file)}.approve.tmp`);
writeFileSync(tmp, s.text);
renameSync(tmp, file);
const fm = parseDocument(s.text).parsed.frontmatter || {};
const record = emit('brief-approved',
{ project: dirname(file), slug: fm.slug ?? null, intent_hash: s.hash },
{ path: join(resolveApprovalDataDir(), STATS_FILENAME) });
process.stdout.write(JSON.stringify({
stamped: true, intent_hash: s.hash, approved_at: s.approvedAt, record,
stamped: true, intent_hash: s.hash, approved_at: s.approvedAt,
}) + '\n');
process.exit(0);
}

5
lib/validators/query-privacy-gate.mjs
View file

@ -4,14 +4,13 @@
// bounded Phase 5 loop turns) — the existing single-pass Phase 5 path is
// unchanged (Step 6, plan-v2).
//
// Two-tier, same shape as lib/exporters/endpoint-validator.mjs's SSRF gate:
// Two-tier:
// - WARN tier — absolute filesystem paths, repo-internal identifiers.
// Operator-overridable via `strict: false` / `--soft` (matches
// lib/validators/research-validator.mjs's strict/soft convention), and
// fully bypassable via the VOYAGE_QUERY_PRIVACY_ALLOW=1 opt-in.
// - HARD-BLOCK tier — secret-shaped tokens. NEVER overridable by strict,
// --soft, or the opt-in env var — mirrors endpoint-validator.mjs's
// HARD_BLOCKED_HOSTS, where an opt-in widens the warn tier but never
// --soft, or the opt-in env var: an opt-in widens the warn tier but never
// unlocks the permanently-blocked one.
//
// CLI shim:

342
scripts/comparison-gate.mjs
View file

@ -1,342 +0,0 @@
#!/usr/bin/env node
// scripts/comparison-gate.mjs
// The comparison gate - the lifted target in PLAN.md (operator, 2026-09-27): Voyage + the
// operator's way of working is level with or ahead of the related frameworks in every area,
// and every Voyage part a built-in Claude Code primitive covers is removed or has a measured
// reason to stay. Two counts, each with a denominator read from its SOURCE:
//
// comparison areas where no reference is ahead (the areas: the first column of the table
// under `source.heading` in `source.loc`;
// counted = areas where every reference
// cell's verdict is `likt` or `foran`)
// tidy covered parts that are resolved (the parts: `git ls-files` over PART_SCOPE,
// one part per eval case; counted = parts a
// primitive covers that are `fjernet` or
// carry a `maalt-grunn` with evidence)
//
// It fells the table on MISSING EVIDENCE, never on a verdict it disagrees with: a cited file
// must exist inside its root and hold the cited lines, a pinned clone must sit at its pin, a
// web citation needs a verbatim quote, and an absence needs zero hits plus a known-positive
// control. A verdict is a reader's judgement; green also requires `reader.kind: uavhengig`.
//
// Exit 0 = both counts full, no evidence problem, independent reader; 1 = anything short of
// that, including a table that does not exist (NOT MEASURED, never 0); 2 = usage error.
// It only READS. The table is local-only (docs/comparison.local.json, `*.local.*`).
//
// Usage: node scripts/comparison-gate.mjs [--json] [--data <file>] [--parts-from <repo>]
import { readFileSync, existsSync, statSync } from 'node:fs';
import { homedir } from 'node:os';
import { join, resolve, dirname, relative, isAbsolute, sep } from 'node:path';
import { fileURLToPath } from 'node:url';
import { execFileSync } from 'node:child_process';
import { isMain } from '../lib/util/is-main.mjs';
const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
export const DEFAULT_DATA = join(REPO_ROOT, 'docs', 'comparison.local.json');
export const PART_SCOPE = ['commands', 'agents', 'hooks', 'lib', 'scripts', 'templates', 'evals', 'settings.json', 'verify.sh'];
export const VERDICTS = ['foran', 'likt', 'bak'];
const LEVEL_OR_AHEAD = new Set(['foran', 'likt']);
export const COVERAGE = ['dekket', 'delvis', 'ikke-dekket'];
export const RESOLUTIONS = ['maalt-grunn', 'aapen', 'fjernet'];
const toPosix = (p) => p.split(sep).join('/');
const nonEmpty = (s) => typeof s === 'string' && s.trim() !== '';
// `ROOT:path:N` or `ROOT:path:N-M`, lines counted from 1. Anything else is not a loc.
export function parseLoc(text) {
const m = /^([A-Z]{2,}):(.+):(\d+)(?:-(\d+))?$/.exec(String(text ?? ''));
if (!m) return null;
const start = Number(m[3]);
const end = m[4] === undefined ? start : Number(m[4]);
if (start < 1 || end < start) return null;
return { root: m[1], path: m[2], start, end };
}
// The first column of the first table after `heading`, header and separator rows excluded.
export function parseSourceAreas(markdown, heading) {
const all = String(markdown).split('\n');
const at = all.findIndex((l) => l.trim() === heading);
if (at === -1) return null;
const rows = [];
let inTable = false;
for (const line of all.slice(at + 1)) {
if (/^#{1,6}\s/.test(line)) break;
if (line.trim().startsWith('|')) {
inTable = true;
rows.push(line);
} else if (inTable) break;
}
return rows.slice(2).map((r) => r.split('|')[1].trim()).filter(Boolean);
}
// One part per tracked file in PART_SCOPE, except eval cases: one part per `evals/<case>/`.
export function listParts(root) {
const out = execFileSync('git', ['ls-files', '-z', '--', ...PART_SCOPE], { cwd: root, encoding: 'utf-8' });
const parts = new Set();
for (const f of out.split('\0').filter(Boolean)) {
const segs = f.split('/');
parts.add(segs[0] === 'evals' && segs.length > 2 ? `evals/${segs[1]}/` : f);
}
return [...parts].sort();
}
function resolveRoot(value) {
if (!nonEmpty(value)) return null;
if (value === '~' || value.startsWith('~/')) return join(homedir(), value.slice(2));
return isAbsolute(value) ? value : resolve(REPO_ROOT, value);
}
function lineCount(file) {
const text = readFileSync(file, 'utf-8');
if (text === '') return 0;
return text.split('\n').length - (text.endsWith('\n') ? 1 : 0);
}
// Problems with one evidence item, each prefixed with `where`.
function checkEvidence(item, where, roots) {
if (!item || typeof item !== 'object') return [`${where}: an evidence item must be an object`];
const problems = [];
if (!nonEmpty(item.says)) problems.push(`${where}: evidence needs \`says\``);
if ('loc' in item) {
const loc = parseLoc(item.loc);
if (!loc) return [...problems, `${where}: ${item.loc} is not ROOT:path:N[-M]`];
const root = roots[loc.root];
if (!root) return [...problems, `${where}: ${item.loc} names an unknown root ${loc.root}`];
const file = resolve(root, loc.path);
const rel = relative(root, file);
if (rel.startsWith('..') || isAbsolute(rel)) return [...problems, `${where}: ${item.loc} points outside its root`];
if (!existsSync(file) || !statSync(file).isFile()) return [...problems, `${where}: ${item.loc} does not exist`];
const n = lineCount(file);
if (loc.end > n) problems.push(`${where}: ${item.loc} cites past the end of a file of ${n} lines`);
return problems;
}
if ('url' in item) {
if (!/^https:\/\/\S+$/.test(String(item.url))) problems.push(`${where}: ${item.url} is not an https URL`);
if (!nonEmpty(item.quote)) problems.push(`${where}: a web citation needs a verbatim \`quote\``);
return problems;
}
if ('absence' in item) {
const a = item.absence ?? {};
if (!nonEmpty(a.query)) problems.push(`${where}: an absence needs the \`query\` that was run`);
if (a.hits !== 0) problems.push(`${where}: an absence must report hits: 0, got ${JSON.stringify(a.hits)}`);
if (!nonEmpty(a.known_positive)) problems.push(`${where}: an absence needs a \`known_positive\` control`);
return problems;
}
return [...problems, `${where}: evidence must be one of loc, url or absence`];
}
function checkEvidenceList(list, where, roots) {
if (!Array.isArray(list) || list.length === 0) return [`${where}: no evidence`];
return list.flatMap((item, i) => checkEvidence(item, `${where}[${i}]`, roots));
}
function pinProblems(references, roots) {
const problems = [];
for (const ref of references) {
if (!nonEmpty(ref.pin)) continue;
const root = roots[ref.id];
if (!root || !existsSync(join(root, '.git'))) {
problems.push(`reference ${ref.id}: pinned ${ref.pin} but ${root ?? 'no root'} is not a git clone`);
continue;
}
let head = '';
try {
head = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: root, encoding: 'utf-8' }).trim();
} catch { /* reported below */ }
if (!head.startsWith(ref.pin)) problems.push(`reference ${ref.id}: pinned ${ref.pin} but the clone's HEAD is ${head.slice(0, 7) || 'unreadable'}`);
}
return problems;
}
function evaluateComparison(data, roots) {
const problems = [];
const references = Array.isArray(data.references) ? data.references : [];
if (references.length === 0) problems.push('no references to compare against');
problems.push(...pinProblems(references, roots));
let sourceAreas = null;
const src = parseLoc(`${data.source?.loc ?? ''}:1`);
const srcFile = src && roots[src.root] ? resolve(roots[src.root], src.path) : null;
if (!srcFile || !existsSync(srcFile)) {
problems.push(`source ${data.source?.loc ?? '(none)'}: not readable - the denominator cannot be read`);
} else {
sourceAreas = parseSourceAreas(readFileSync(srcFile, 'utf-8'), data.source.heading);
if (!sourceAreas || sourceAreas.length === 0) {
problems.push(`source ${data.source.loc}: no table under ${JSON.stringify(data.source.heading)}`);
sourceAreas = null;
}
}
const areas = Array.isArray(data.areas) ? data.areas : [];
const byTitle = new Map(areas.map((a) => [a.title, a]));
const counted = [];
const behind = [];
const perReference = Object.fromEntries(references.map((r) => [r.id, 0]));
const contested = [];
for (const title of sourceAreas ?? []) {
if (!byTitle.has(title)) problems.push(`area ${title}: in the source but missing from the table`);
}
for (const area of areas) {
const id = area.id ?? area.title;
if (sourceAreas && !sourceAreas.includes(area.title)) {
problems.push(`area ${area.title}: not in the source - not counted`);
continue;
}
problems.push(...checkEvidenceList(area.own?.evidence, `${id} own`, roots));
const behindRefs = [];
for (const ref of references) {
const cell = area.cells?.[ref.id];
if (!cell) { problems.push(`${id} ${ref.id}: cell missing`); continue; }
if (!VERDICTS.includes(cell.verdict)) problems.push(`${id} ${ref.id}: verdict ${JSON.stringify(cell.verdict)} is not one of ${VERDICTS.join('|')}`);
if (!nonEmpty(cell.why)) problems.push(`${id} ${ref.id}: a verdict needs \`why\``);
problems.push(...checkEvidenceList(cell.evidence, `${id} ${ref.id}`, roots));
if (cell.contested) contested.push(`${id} (${ref.id})`);
if (!LEVEL_OR_AHEAD.has(cell.verdict)) behindRefs.push(ref.id);
else perReference[ref.id]++;
}
if (behindRefs.length) behind.push({ area: id, references: behindRefs });
else counted.push(id);
}
const total = sourceAreas ? sourceAreas.length : null;
return {
problems,
comparison: {
total,
level_or_ahead: total === null ? null : counted.length,
behind,
per_reference: perReference,
contested,
references: references.map((r) => `${r.id}${r.pin ? `@${r.pin}` : ''} ${r.name ?? ''}`.trim()),
},
};
}
function evaluateTidy(tidy, parts, roots) {
const problems = [];
if (!tidy || typeof tidy !== 'object') {
return { problems: ['tidy: missing - the tidy denominator is not measured'], tidy: { parts: parts.length, covered: null, resolved: null } };
}
const primitives = Array.isArray(tidy.primitives) ? tidy.primitives : [];
const known = new Set();
for (const p of primitives) {
known.add(p.id);
if (!/^https:\/\/\S+$/.test(String(p.doc ?? ''))) problems.push(`primitive ${p.id}: needs a documentation URL`);
if (!nonEmpty(p.quote)) problems.push(`primitive ${p.id}: needs a verbatim \`quote\` from its documentation`);
}
const listed = Array.isArray(tidy.parts) ? tidy.parts : [];
const tree = new Set(parts);
const seen = new Set();
let covered = 0;
let resolved = 0;
for (const part of listed) {
const where = `part ${part.path}`;
if (seen.has(part.path)) problems.push(`${where}: listed twice`);
seen.add(part.path);
if (!COVERAGE.includes(part.coverage)) { problems.push(`${where}: coverage ${JSON.stringify(part.coverage)} is not one of ${COVERAGE.join('|')}`); continue; }
if (part.resolution !== 'fjernet' && !tree.has(part.path)) problems.push(`${where}: not in the tree and not marked fjernet`);
if (part.coverage === 'ikke-dekket') continue;
covered++;
const by = Array.isArray(part.covered_by) ? part.covered_by : [];
if (by.length === 0) problems.push(`${where}: ${part.coverage} needs \`covered_by\``);
for (const id of by) if (!known.has(id)) problems.push(`${where}: covered by an unknown primitive ${id}`);
if (!RESOLUTIONS.includes(part.resolution)) {
problems.push(`${where}: resolution ${JSON.stringify(part.resolution)} is not one of ${RESOLUTIONS.join('|')}`);
} else if (part.resolution === 'maalt-grunn') {
if (!part.reason) problems.push(`${where}: maalt-grunn needs a \`reason\` with evidence`);
else {
const p = checkEvidence(part.reason, `${where} reason`, roots);
problems.push(...p);
if (p.length === 0) resolved++;
}
} else if (part.resolution === 'fjernet') {
if (tree.has(part.path)) problems.push(`${where}: marked fjernet but still in the tree`);
else if (!nonEmpty(part.commit)) problems.push(`${where}: fjernet needs the \`commit\` that removed it`);
else resolved++;
}
}
for (const p of parts) if (!seen.has(p)) problems.push(`part ${p}: in the tree but missing from the tidy list`);
return { problems, tidy: { parts: parts.length, covered, resolved, primitives: primitives.length } };
}
export function evaluate(data, { parts = [] } = {}) {
const roots = {};
for (const [id, value] of Object.entries(data?.roots ?? {})) roots[id] = resolveRoot(value);
const cmp = evaluateComparison(data ?? {}, roots);
const tdy = evaluateTidy(data?.tidy, parts, roots);
const problems = [...cmp.problems, ...tdy.problems];
const reader = data?.reader ?? {};
const reasons = [];
if (problems.length) reasons.push(`${problems.length} evidence problem(s)`);
const c = cmp.comparison;
if (c.total === null) reasons.push('the comparison denominator is not measured');
else if (c.level_or_ahead < c.total) reasons.push(`${c.level_or_ahead} of ${c.total} areas level or ahead`);
if (reader.kind !== 'uavhengig') reasons.push(`the verdicts are by ${reader.kind ?? 'nobody'}, not an independent reader`);
if (tdy.tidy.covered === null) reasons.push('the tidy list is not measured');
else if (tdy.tidy.resolved < tdy.tidy.covered) reasons.push(`${tdy.tidy.resolved} of ${tdy.tidy.covered} covered parts resolved`);
return {
measured: true,
reader,
comparison: c,
tidy: tdy.tidy,
problems,
reasons,
green: reasons.length === 0,
};
}
export function render(r) {
if (!r.measured) return `Comparison gate: NOT MEASURED - ${r.reasons.join('; ')}\nRED`;
const c = r.comparison;
const out = [];
out.push(`Comparison gate - areas where no reference is ahead of Voyage + the operator's way of working`);
out.push(` references: ${c.references.join(' · ')}`);
out.push(` level or ahead: ${c.level_or_ahead ?? 'NOT MEASURED'} of ${c.total ?? '?'}`);
for (const b of c.behind) out.push(` behind: ${b.area} (${b.references.join(', ')})`);
out.push(` per reference: ${Object.entries(c.per_reference).map(([id, n]) => `${id}: ${n} of ${c.total ?? '?'}`).join(' · ')}`);
if (c.contested.length) out.push(` contested cells: ${c.contested.join(', ')}`);
out.push(` reader: ${r.reader.kind ?? 'none'}${r.reader.who ? ` - ${r.reader.who}` : ''}${r.reader.date ? ` (${r.reader.date})` : ''}`
+ `${r.reader.kind === 'uavhengig' ? '' : ' - green requires an independent reader'}`);
const t = r.tidy;
out.push(`Tidy - Voyage parts a documented Claude Code primitive covers`);
out.push(` parts in the tree: ${t.parts} · covered: ${t.covered ?? 'NOT MEASURED'} · resolved: ${t.resolved ?? '-'} of ${t.covered ?? '-'}`);
out.push(`Evidence problems: ${r.problems.length}`);
for (const p of r.problems) out.push(` - ${p}`);
out.push(r.green ? 'GREEN' : `RED - ${r.reasons.join('; ')}`);
return out.join('\n');
}
export function main(argv) {
let json = false;
let dataFile = DEFAULT_DATA;
let partsFrom = REPO_ROOT;
const usage = 'usage: comparison-gate.mjs [--json] [--data <file>] [--parts-from <repo>]';
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
const value = argv[i + 1];
if (a === '--json') { json = true; continue; }
if (a === '--data' && value !== undefined) { dataFile = resolve(value); i++; continue; }
if (a === '--parts-from' && value !== undefined) { partsFrom = resolve(value); i++; continue; }
process.stderr.write(`comparison-gate: unknown argument ${a}\n${usage}\n`);
return 2;
}
let result;
if (!existsSync(dataFile)) {
result = { measured: false, green: false, reasons: [`${toPosix(relative(REPO_ROOT, dataFile))} does not exist`] };
} else {
let data = null;
try {
data = JSON.parse(readFileSync(dataFile, 'utf-8'));
} catch (err) {
result = { measured: false, green: false, reasons: [`${dataFile} is not valid JSON: ${err.message}`] };
}
if (data) result = evaluate(data, { parts: listParts(partsFrom) });
}
process.stdout.write(`${json ? JSON.stringify(result, null, 2) : render(result)}\n`);
return result.green ? 0 : 1;
}
if (isMain(import.meta.url)) {
process.exitCode = main(process.argv.slice(2));
}

585
scripts/end-state-gate.mjs
View file

@ -1,585 +0,0 @@
#!/usr/bin/env node
// scripts/end-state-gate.mjs
// End-state gate — counts the distance to "Voyage is finished" as five tallies:
//
// defects open pipeline defects (listed in the registry, each with a runnable check)
// experiments unresolved experiments/opt-ins (listed in the registry, each with a runnable check)
// dormant-agents spawnable agents no command spawns (measured structurally, see spawnedNames)
// decisions open operator decisions (counted in STATE.md by a fixed marker)
// feat-after-freeze feat commits after the freeze tag (measured from git)
//
// "Finished" = all five at 0 AND the registry is intact against its frozen denominator
// (tests/fixtures/end-state-frozen.json: an entry may close, never disappear or change its
// check). Exit 0 = green, 1 = red (a tally > 0, a tally not measurable, or the registry
// tampered with), 2 = usage or registry error.
//
// A registry entry is OPEN while ALL of its conditions hold. An entry whose check is null,
// or whose check cannot run (missing file, empty glob, missing section), is NOT FELLABLE and
// is counted as open — a check that cannot fire must never read as "fixed".
//
// Usage: node scripts/end-state-gate.mjs [--json] [--root <dir>]
import {
readFileSync, writeFileSync, existsSync, readdirSync, statSync, realpathSync,
mkdirSync, mkdtempSync, symlinkSync, rmSync,
} from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve, dirname, relative, sep, basename } from 'node:path';
import { fileURLToPath } from 'node:url';
import { createHash } from 'node:crypto';
import { spawnSync } from 'node:child_process';
import { isMain } from '../lib/util/is-main.mjs';
export const REGISTRY_PATH = 'scripts/end-state-registry.json';
export const FROZEN_PATH = 'tests/fixtures/end-state-frozen.json';
const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
const toPosix = (p) => p.split(sep).join('/');
function walk(dir) {
const out = [];
for (const name of readdirSync(dir)) {
const p = join(dir, name);
if (statSync(p).isDirectory()) out.push(...walk(p));
else out.push(p);
}
return out;
}
// Supported path forms: an exact file, `dir/*.ext`, `dir/**/*.ext`.
export function expandPath(root, pattern) {
if (!pattern.includes('*')) {
const abs = join(root, pattern);
if (!existsSync(abs)) throw new Error(`path not found: ${pattern}`);
return [abs];
}
const m = pattern.match(/^(.+?)\/(\*\*\/)?\*(\.[A-Za-z0-9]+)$/);
if (!m) throw new Error(`unsupported glob: ${pattern}`);
const [, base, deep, ext] = m;
const absBase = join(root, base);
if (!existsSync(absBase) || !statSync(absBase).isDirectory()) {
throw new Error(`glob base not found: ${pattern}`);
}
const files = deep
? walk(absBase)
: readdirSync(absBase).map((f) => join(absBase, f)).filter((f) => statSync(f).isFile());
const hits = files.filter((f) => f.endsWith(ext)).sort();
if (hits.length === 0) throw new Error(`glob matched no files: ${pattern}`);
return hits;
}
// The text from the first line starting with `heading` up to the next `## ` heading.
function sectionOf(text, heading, label) {
const lines = text.split('\n');
const start = lines.findIndex((l) => l.startsWith(heading));
if (start === -1) throw new Error(`section "${heading}" not found in ${label}`);
let end = lines.length;
for (let i = start + 1; i < lines.length; i++) {
if (lines[i].startsWith('## ')) { end = i; break; }
}
return lines.slice(start, end).join('\n');
}
export const TEST_TIMEOUT_MS = 120_000;
const escapeRegExp = (t) => String(t).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
// Every `export` name a module declares, so a stub can offer the same surface
// and none of the behaviour.
const EXPORT_NAME = /^\s*export\s+(?:async\s+)?(?:function\*?|const|let|var|class)\s+([A-Za-z_$][\w$]*)/gm;
/**
* A stub of a module: the same export names, every one of them inert.
*
* This is the mutant a behaviour probe must FELL. A test that still passes
* when the module it names does nothing is not measuring that module — the
* empty test that closed D-03 and D-04 on a tree without `lib/verification/`
* is the limiting case (measured 2026-09-18).
*/
export function stubModule(source) {
const names = new Set();
for (const m of String(source).matchAll(EXPORT_NAME)) names.add(m[1]);
const lines = [...names].map((n) => `export const ${n} = undefined;`);
if (/^\s*export\s+default\b/m.test(source)) lines.push('export default undefined;');
return `${lines.join('\n')}\n`;
}
// Build `dest` as a shallow copy of `root`: every entry is a SYMLINK to the
// original, except the files in `files`, which are written for real (and whose
// parent directories are therefore real directories too). Node resolves an
// ESM import through the realpath, so a symlinked test file would import the
// ORIGINAL module and never see the stub — only the files that must differ are
// materialised. Nothing is ever written inside the measured tree.
function overlay(root, dest, files) {
const realDirs = new Set();
for (const rel of files.keys()) {
const parts = rel.split('/');
for (let i = 0; i < parts.length - 1; i++) realDirs.add(parts.slice(0, i + 1).join('/'));
}
const build = (relDir) => {
const absSrc = relDir === '' ? root : join(root, relDir);
const absDst = relDir === '' ? dest : join(dest, relDir);
mkdirSync(absDst, { recursive: true });
if (!existsSync(absSrc)) return;
for (const name of readdirSync(absSrc)) {
const childRel = relDir === '' ? name : `${relDir}/${name}`;
if (realDirs.has(childRel)) { build(childRel); continue; }
if (files.has(childRel)) continue;
symlinkSync(join(absSrc, name), join(absDst, name));
}
};
build('');
for (const [rel, body] of files) writeFileSync(join(dest, rel), body);
}
// Run ONE named test out of one test file and report whether it passed.
// `null` = the runner produced no TAP line for that name (it did not run at
// all), which is never a pass.
function runNamedTest(root, testRel, name) {
// The gate itself often runs UNDER the test runner (the gate's own test
// spawns it). `NODE_TEST_CONTEXT` inherited into this child makes it report
// over the parent's IPC channel instead of stdout, and the TAP line would
// never arrive — a probe that silently stops felling anything.
const env = { ...process.env };
for (const key of Object.keys(env)) {
if (key.startsWith('NODE_TEST_')) delete env[key];
}
const r = spawnSync(
process.execPath,
['--test', '--test-reporter=tap', `--test-name-pattern=^${escapeRegExp(name)}$`, testRel],
{ cwd: root, encoding: 'utf8', timeout: TEST_TIMEOUT_MS, env },
);
if (r.error) throw new Error(`test runner did not run ${testRel}: ${r.error.message}`);
// The file itself reports as a subtest too, so the summary counters are
// ambiguous; the TAP line carrying the exact name is not. TAP escapes `#`
// and `\`, so unescape before comparing.
for (const line of String(r.stdout ?? '').split('\n')) {
const m = line.match(/^\s*(not )?ok \d+ - (.*)$/);
if (!m) continue;
if (m[2].replace(/\\(.)/g, '$1').trim() !== name) continue;
return m[1] === undefined;
}
return null;
}
/**
* A condition that RUNS a named test, reads its result, and requires that the
* same test FELLS a stub of the module it claims to bind.
*
* A grep condition can only ask whether a string is present, so a "behaviour"
* probe built from one is a phrase probe pointed at code: measured
* 2026-09-18, a two-line file exporting `runPlanVerification` and
* `formatCriteriaEvidence` closed D-03 and D-04 on a tree where nothing was
* fixed. Running a NAMED test closed that hole and opened a smaller one one
* level up: two EMPTY tests with the right names closed them again. A name is
* always forgeable; felling a mutant is not. So the named test is run twice —
* once on the tree, once against a stub of each module in `stubs` — and it
* must not pass the second time.
*
* `expect: "passes"` holds when the named test passes; `"fails"` holds when it
* does not. A test file that is missing, a name that matches nothing, a runner
* that cannot start, a missing or unfelled stub target: all THROW, which the
* caller counts as NOT FELLABLE and therefore open. A check that cannot fire
* is never "fixed".
*/
export function evaluateTestCondition(root, cond) {
if (cond.expect !== 'passes' && cond.expect !== 'fails') {
throw new Error(`a test condition's expect must be "passes" or "fails", got ${JSON.stringify(cond.expect)}`);
}
if (typeof cond.name !== 'string' || cond.name.trim() === '') {
throw new Error('a test condition must name exactly one test');
}
if (!Array.isArray(cond.stubs) || cond.stubs.length === 0) {
throw new Error('a test condition must declare `stubs`: the module(s) whose stub the named test must fell');
}
const file = join(root, cond.test);
if (!existsSync(file)) throw new Error(`test file not found: ${cond.test}`);
for (const target of cond.stubs) {
const abs = join(root, target);
if (!existsSync(abs)) throw new Error(`stub target not found: ${target}`);
const sandbox = mkdtempSync(join(tmpdir(), 'end-state-mutant-'));
try {
overlay(root, sandbox, new Map([
[cond.test, readFileSync(file, 'utf8')],
[target, stubModule(readFileSync(abs, 'utf8'))],
]));
if (runNamedTest(sandbox, cond.test, cond.name) === true) {
throw new Error(
`"${cond.name}" still passes against a stub of ${target} — it binds the name, not the behaviour`,
);
}
} finally {
// Only ever a directory this function made, under the system temp dir.
if (sandbox.startsWith(realpathSync(tmpdir()))) rmSync(sandbox, { recursive: true, force: true });
}
}
const passed = runNamedTest(root, cond.test, cond.name);
if (passed === null) throw new Error(`no test named "${cond.name}" ran in ${cond.test}`);
return cond.expect === 'passes' ? passed : !passed;
}
export function evaluateCondition(root, cond) {
if (cond.test !== undefined) return evaluateTestCondition(root, cond);
if (cond.expect !== 'match' && cond.expect !== 'no-match') {
throw new Error(`expect must be "match" or "no-match", got ${JSON.stringify(cond.expect)}`);
}
if (String(cond.flags ?? '').includes('g')) throw new Error('the g flag is not allowed');
const paths = Array.isArray(cond.path) ? cond.path : [cond.path];
const files = paths.flatMap((p) => expandPath(root, p));
const anyMatch = files.some((f) => {
let text = readFileSync(f, 'utf8');
if (cond.section) text = sectionOf(text, cond.section, toPosix(relative(root, f)));
return new RegExp(cond.pattern, cond.flags ?? '').test(text);
});
return cond.expect === 'match' ? anyMatch : !anyMatch;
}
export function evaluateCheck(root, check) {
if (check === null || check === undefined) {
return { status: 'not-fellable', detail: 'no runnable check' };
}
if (!Array.isArray(check) || check.length === 0) {
return { status: 'not-fellable', detail: 'check must be a non-empty list of conditions' };
}
try {
// Evaluate every condition (no short-circuit) so a broken one is always reported.
const results = check.map((c) => evaluateCondition(root, c));
return { status: results.every(Boolean) ? 'open' : 'closed', detail: '' };
} catch (err) {
return { status: 'not-fellable', detail: err.message };
}
}
// --- the frozen denominator -------------------------------------------------
// JSON with object keys sorted, so a signature does not depend on key order.
function canonical(value) {
if (Array.isArray(value)) return `[${value.map(canonical).join(',')}]`;
if (value && typeof value === 'object') {
return `{${Object.keys(value).sort().map((k) => `${JSON.stringify(k)}:${canonical(value[k])}`).join(',')}}`;
}
return JSON.stringify(value ?? null);
}
export function checkSignature(check) {
return createHash('sha256').update(canonical(check)).digest('hex');
}
export function loadFrozen(root) {
return JSON.parse(readFileSync(join(root, FROZEN_PATH), 'utf8'));
}
// The judged party must not own the ledger: every frozen entry must still be present with
// the same check, and the counting configuration must not move. New entries are allowed.
export function verifyIntegrity(registry, frozen) {
const violations = [];
for (const kind of ['defects', 'experiments']) {
const byId = new Map();
for (const e of registry[kind] || []) {
if (byId.has(e.id)) violations.push(`${kind}: duplicate id ${e.id}`);
byId.set(e.id, e);
}
for (const [id, sig] of Object.entries(frozen[kind] || {})) {
const e = byId.get(id);
if (!e) {
violations.push(`${kind}: ${id} is frozen but missing from the registry (an entry may close, never disappear)`);
} else if (checkSignature(e.check) !== sig) {
violations.push(`${kind}: ${id} has a changed check (signature ${checkSignature(e.check).slice(0, 12)}, frozen ${sig.slice(0, 12)})`);
}
}
}
for (const key of ['agents', 'decisions', 'freeze']) {
if (canonical(registry[key]) !== canonical(frozen[key])) {
violations.push(`${key}: configuration differs from the frozen manifest`);
}
}
const sites = [].concat(registry.agents?.spawnSites ?? []);
if (sites.some((s) => /(^|\/)agents\//.test(String(s)))) {
violations.push('agents: spawnSites may never include agents/ (an agent file would reference itself)');
}
return { ok: violations.length === 0, violations };
}
// --- tallies ----------------------------------------------------------------
const notMeasured = (detail) => ({ open: null, total: null, items: [], detail });
// What each probe kind can and cannot prove. An entry declares its kind; the
// gate states the limit out loud, because a closed check is only ever as strong
// as the thing it reads. Every kind a registry entry may declare belongs here —
// the entry-shape test rejects a kind the gate cannot explain.
// What no probe in this file can prove. /trekexecute and /trekreview are
// markdown a model interprets, so no unit test can observe that Phase 7 or
// Phase 4.5 was performed. The gate says this out loud rather than letting a
// green defect row imply it.
export const WIRING_NOTE =
'wiring: a behaviour probe proves the capability WORKS. That /trekexecute Phase 7 and ' +
'/trekreview Phase 4.5 actually CALL it is pinned by TEXT in tests/lib/doc-consistency.test.mjs, ' +
'not proven deterministically - real proof is a headless plugin-eval run against a fixture plan (week 40).';
export const PROBE_NOTES = {
phrase:
'a phrase probe closes on rewording: a closed phrase probe is evidence, not proof of behaviour',
behaviour:
'a behaviour probe RUNS a named test AND re-runs it against a stub of the module it binds, which it must fell: neither a stub that exports the right symbols nor an empty test with the right name closes it; it proves the capability WORKS, not that a prose phase calls it',
byte:
"a byte probe reads the file's bytes, so it closes only on a real change to them",
};
function tallyEntries(root, entries) {
const items = entries.map((e) => ({ id: e.id, summary: e.summary, probe: e.probe, ...evaluateCheck(root, e.check) }));
const kinds = Object.keys(PROBE_NOTES).filter((k) => items.some((i) => i.probe === k));
return {
open: items.filter((i) => i.status !== 'closed').length,
total: items.length,
items,
detail: kinds.map((k) => PROBE_NOTES[k]).join('; '),
};
}
// Lines that can carry an instruction: HTML comments and fenced code are blanked (line numbers
// are kept so "the next line" still means the next line). Block quotes need no stripping: every
// spawn form below is anchored at the start of the line, and a quoted line starts with `>`.
function instructionLines(markdown) {
const uncommented = markdown.replace(/<!--[\s\S]*?-->/g, (m) => m.replace(/[^\n]/g, ''));
const out = [];
let fenced = false;
for (const line of uncommented.split('\n')) {
if (/^\s*(```|~~~)/.test(line)) { fenced = !fenced; out.push(''); continue; }
out.push(fenced ? '' : line);
}
return out;
}
const NAME = '[a-z0-9][a-z0-9-]*';
const ROSTER_ROW = new RegExp(`^\\s*\\|\\s*\`(${NAME})\`\\s*\\|`);
const LAUNCH_LINE = /^\s*(?:[-*]\s+|\d+\.\s+)?(?:\*\*)?(?:launch|spawn)\b(?:\*\*)?(.*)$/i;
const LAUNCH_NAME = new RegExp(`\`(?:voyage:)?(${NAME})\`|\\*\\*(${NAME})\\*\\*|voyage:(${NAME})`, 'g');
const AGENT_BLOCK = new RegExp(`^\\s*\\*\\*(${NAME})\\*\\*`);
// The agent names a command markdown file actually spawns. A spawn instruction is one of:
// 1. a row of a table whose header's first cell is `Agent` (a roster the step launches);
// 2. an imperative line that STARTS with Launch/Spawn and names the agent as `name`,
// **name** or voyage:name;
// 3. a line starting with **name** whose next non-empty line starts with `Prompt:`.
// Prose, negations, HTML comments, fenced code and block quotes never count.
export function spawnedNames(markdown) {
const lines = instructionLines(markdown);
const names = new Set();
let inTable = false;
let roster = false;
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
if (/^\s*\|/.test(line)) {
if (!inTable) {
inTable = true;
roster = /^\s*\|\s*Agent\s*\|/.test(line);
continue;
}
const row = roster && line.match(ROSTER_ROW);
if (row) names.add(row[1]);
continue;
}
inTable = false;
const launch = line.match(LAUNCH_LINE);
if (launch) {
for (const m of launch[1].matchAll(LAUNCH_NAME)) names.add(m[1] || m[2] || m[3]);
continue;
}
const block = line.match(AGENT_BLOCK);
if (block) {
let j = i + 1;
while (j < lines.length && lines[j].trim() === '') j++;
if (j < lines.length && /^\s*Prompt:/.test(lines[j])) names.add(block[1]);
}
}
return names;
}
function tallyDormantAgents(root, cfg) {
try {
const agentFiles = expandPath(root, `${cfg.dir}/*.md`);
const spawned = new Set();
for (const f of expandPath(root, cfg.spawnSites)) {
for (const n of spawnedNames(readFileSync(f, 'utf8'))) spawned.add(n);
}
const items = [];
for (const file of agentFiles) {
const text = readFileSync(file, 'utf8');
const fm = text.startsWith('---') ? text.split('\n---')[0] : '';
if (cfg.referenceMarker && fm.includes(cfg.referenceMarker)) continue;
const name = (fm.match(/^name:\s*(\S+)\s*$/m) || [])[1] || basename(file, '.md');
const isSpawned = spawned.has(name);
items.push({
id: name,
summary: toPosix(relative(root, file)),
status: isSpawned ? 'closed' : 'open',
detail: isSpawned ? '' : `no spawn instruction in ${cfg.spawnSites}`,
});
}
return {
open: items.filter((i) => i.status === 'open').length,
total: items.length,
items,
detail: 'spawnable = agent files without the reference-document marker; spawned = named in a spawn instruction (an `Agent` table row, a line starting with Launch/Spawn, or a **name** block followed by Prompt:); prose, comments, fences and quotes do not count',
};
} catch (err) {
return notMeasured(`not measurable: ${err.message}`);
}
}
function tallyDecisions(root, cfg) {
const abs = join(root, cfg.file);
if (!existsSync(abs)) {
return notMeasured(`not measurable: ${cfg.file} not found (it is local-only)`);
}
try {
const section = sectionOf(readFileSync(abs, 'utf8'), cfg.section, cfg.file);
const lines = section.split('\n');
const openRe = new RegExp(cfg.open);
const closedRe = new RegExp(cfg.closed);
const open = lines.filter((l) => openRe.test(l));
const closed = lines.filter((l) => closedRe.test(l));
// Format drift guard: a list item that carries neither marker would silently read as "0 open".
const unmarked = lines.filter((l) => /^\s*(\d+\.|[-*]) /.test(l) && !openRe.test(l) && !closedRe.test(l));
if (unmarked.length > 0) {
return notMeasured(`not measurable: ${unmarked.length} unmarked list item(s) in ${cfg.section} — mark each with the open/decided marker`);
}
return {
open: open.length,
total: open.length + closed.length,
items: open.map((l, i) => ({ id: `#${i + 1}`, summary: l.replace(openRe, '').slice(0, 100), status: 'open', detail: '' })),
detail: `marker: open = /${cfg.open}/, decided = /${cfg.closed}/; this counts ticks — a ticked line is not verified to be an actual decision`,
};
} catch (err) {
return notMeasured(`not measurable: ${err.message}`);
}
}
function tallyFeatAfterFreeze(root, cfg) {
const git = (...args) => spawnSync('git', ['-C', root, ...args], { encoding: 'utf8' });
const top = git('rev-parse', '--show-toplevel');
if (top.status !== 0) return notMeasured(`not measured: ${toPosix(root)} is not a git work tree`);
let atTop = false;
try { atTop = realpathSync(top.stdout.trim()) === realpathSync(root); } catch { atTop = false; }
if (!atTop) return notMeasured('not measured: the gate root is not the top of its git work tree');
const ref = `refs/tags/${cfg.tag}`;
if (git('rev-parse', '-q', '--verify', ref).status !== 0) {
return notMeasured(`not measured: no freeze tag "${cfg.tag}" yet — open until the freeze is declared`);
}
const log = git('log', '--format=%s', `${ref}..HEAD`);
if (log.status !== 0) return notMeasured(`not measured: git log failed: ${log.stderr.trim()}`);
const subjects = log.stdout.split('\n').filter(Boolean);
const featRe = new RegExp(cfg.featPattern);
const feats = subjects.filter((s) => featRe.test(s));
return {
open: feats.length,
total: subjects.length,
items: feats.map((s, i) => ({ id: `#${i + 1}`, summary: s, status: 'open', detail: '' })),
detail: `commits after ${cfg.tag} whose subject matches /${cfg.featPattern}/`,
};
}
export function measure(root, registry, frozen) {
const d = registry.decisions;
const f = registry.freeze;
const rows = [
{ id: 'defects', label: 'open pipeline defects', source: `${REGISTRY_PATH}#defects`, ...tallyEntries(root, registry.defects) },
{ id: 'experiments', label: 'unresolved experiments/opt-ins', source: `${REGISTRY_PATH}#experiments`, ...tallyEntries(root, registry.experiments) },
{ id: 'dormant-agents', label: 'dormant agents', source: `${registry.agents.dir}/*.md vs ${registry.agents.spawnSites}`, ...tallyDormantAgents(root, registry.agents) },
{ id: 'decisions', label: 'open operator decisions', source: `${d.file} ${d.section.replace(/^#+\s*/, '§ ')}`, ...tallyDecisions(root, d) },
{ id: 'feat-after-freeze', label: 'feat commits after the freeze tag', source: `git log ${f.tag}..HEAD`, ...tallyFeatAfterFreeze(root, f) },
].map((r) => ({ ...r, target: 0 }));
let integrity;
try {
integrity = { ...verifyIntegrity(registry, frozen ?? loadFrozen(root)), detail: '' };
} catch (err) {
integrity = { ok: null, violations: [], detail: `not measurable: cannot read ${FROZEN_PATH}: ${err.message}` };
}
return { green: rows.every((r) => r.open === 0) && integrity.ok === true, rows, integrity, wiring: WIRING_NOTE };
}
export function loadRegistry(root) {
const abs = join(root, REGISTRY_PATH);
let reg;
try {
reg = JSON.parse(readFileSync(abs, 'utf8'));
} catch (err) {
throw new Error(`cannot read ${REGISTRY_PATH}: ${err.message}`);
}
for (const key of ['defects', 'experiments']) {
if (!Array.isArray(reg[key])) throw new Error(`${REGISTRY_PATH}: "${key}" must be a list`);
}
for (const key of ['agents', 'decisions', 'freeze']) {
if (!reg[key] || typeof reg[key] !== 'object') throw new Error(`${REGISTRY_PATH}: "${key}" must be an object`);
}
return reg;
}
export function render(result) {
const zero = result.rows.filter((r) => r.open === 0).length;
const out = [];
out.push(`Voyage end-state gate: ${result.green ? 'GREEN' : 'RED'} (${zero} of ${result.rows.length} tallies at 0)`);
const integrity = result.integrity;
if (integrity?.ok === true) {
out.push(`registry integrity: intact against ${FROZEN_PATH}`);
} else if (integrity?.ok === false) {
out.push('registry integrity: VIOLATED — the gate cannot be green while the ledger differs from its frozen denominator');
for (const v of integrity.violations) out.push(` - ${v}`);
} else if (integrity) {
out.push(`registry integrity: n/a — ${integrity.detail}`);
}
out.push(result.wiring ?? WIRING_NOTE);
out.push('');
out.push('| tally | open | of | target | source |');
out.push('|---|---|---|---|---|');
for (const r of result.rows) {
const open = r.open === null ? 'n/a' : String(r.open);
const total = r.total === null ? 'n/a' : String(r.total);
out.push(`| ${r.id} | ${open} | ${total} | ${r.target} | ${r.source} |`);
}
for (const r of result.rows) {
out.push('');
out.push(`${r.id} — ${r.label}${r.detail ? ` (${r.detail})` : ''}`);
for (const i of r.items) {
if (i.status === 'closed') continue;
let tag = i.status === 'not-fellable' ? 'NOT FELLABLE, counted open' : 'open';
if (i.probe) tag += ` · ${i.probe} probe`;
out.push(` [${tag}] ${i.id}: ${i.summary}${i.detail ? ` — ${i.detail}` : ''}`);
}
const closed = r.items.filter((i) => i.status === 'closed' && i.probe);
if (closed.length > 0) {
out.push(` closed: ${closed.map((i) => `${i.id} (${i.probe} probe)`).join(', ')}`);
}
}
return out.join('\n');
}
export function main(argv) {
let json = false;
let root = REPO_ROOT;
for (let i = 0; i < argv.length; i++) {
if (argv[i] === '--json') json = true;
else if (argv[i] === '--root' && argv[i + 1]) root = resolve(argv[++i]);
else {
process.stderr.write(`end-state-gate: unknown argument ${argv[i]}\nusage: end-state-gate.mjs [--json] [--root <dir>]\n`);
return 2;
}
}
let registry;
try {
registry = loadRegistry(root);
} catch (err) {
process.stderr.write(`end-state-gate: ${err.message}\n`);
return 2;
}
const result = measure(root, registry);
process.stdout.write((json ? JSON.stringify(result, null, 2) : render(result)) + '\n');
return result.green ? 0 : 1;
}
if (isMain(import.meta.url)) {
process.exitCode = main(process.argv.slice(2));
}

182
scripts/end-state-registry.json
View file

@ -1,182 +0,0 @@
{
"defects": [
{
"id": "D-01",
"summary": "commands/trekplan.md still references TeamCreate/TeamDelete (allowed-tools and the execute-with-team path); both tools were removed in Claude Code 2.1.178, so that path always falls back to sequential",
"closesWhen": "trekplan.md no longer names TeamCreate or TeamDelete",
"probe": "phrase",
"check": [
{
"path": "commands/trekplan.md",
"pattern": "\\bTeam(Create|Delete)\\b",
"expect": "match"
}
]
},
{
"id": "D-02",
"summary": "commands/trekresearch.md tells the swarm engine to run the `### Bridge agent` block, but that heading was removed together with gemini-bridge",
"closesWhen": "the dangling reference is gone (or the heading exists again)",
"probe": "phrase",
"check": [
{
"path": "commands/trekresearch.md",
"pattern": "`### Bridge agent`",
"expect": "match"
},
{
"path": "commands/trekresearch.md",
"pattern": "^### Bridge agent",
"flags": "m",
"expect": "no-match"
}
]
},
{
"id": "D-03",
"summary": "commands/trekexecute.md never runs a trekplan's `## Verification` (where the brief's success criteria land) on the single-session path: Phase 7 says 'Skip for trekplans', and only the multi-session wave path runs master verification",
"closesWhen": "a named test PROVES the single-session path fells a run on a criterion that does not hold (behaviour probe: the gate RUNS tests/lib/criteria-runner.test.mjs, so a stub that only exports the symbol does not close it, and the named test must itself FELL a stub of lib/verification/criteria-runner.mjs, so an EMPTY test with the right name does not close it either; that Phase 7 CALLS it is pinned by tests/lib/doc-consistency.test.mjs, not by this probe)",
"probe": "behaviour",
"check": [
{
"test": "tests/lib/criteria-runner.test.mjs",
"name": "runPlanVerification: a plan whose success criterion FAILS fells the run",
"expect": "fails",
"stubs": [
"lib/verification/criteria-runner.mjs"
]
}
]
},
{
"id": "D-04",
"summary": "agents/brief-conformance-reviewer.md must judge whether a success criterion's verification command 'exists and passes', but its tools are Read/Glob/Grep, so it cannot run anything",
"closesWhen": "a named test PROVES the evidence block the conformance reviewer judges is BUILT IN CODE from real exit codes (behaviour probe: the gate RUNS tests/lib/criteria-runner.test.mjs, so a stub that only exports the symbol does not close it, and the named test must itself FELL a stub of lib/verification/criteria-runner.mjs, so an EMPTY test with the right name does not close it either; that Phase 4.5 CALLS it is pinned by tests/lib/doc-consistency.test.mjs, not by this probe)",
"probe": "behaviour",
"check": [
{
"test": "tests/lib/criteria-runner.test.mjs",
"name": "formatCriteriaEvidence: one row per criterion, with command and exit code",
"expect": "fails",
"stubs": [
"lib/verification/criteria-runner.mjs"
]
}
]
},
{
"id": "D-05",
"summary": "commands/trekplan.md allowed-tools lists TaskCreate/TaskUpdate, which Claude Code 2.1.233 no longer offers on Opus 4.8 / Sonnet 5 / Fable 5 and newer by default (runtime effect not measured)",
"closesWhen": "trekplan.md allowed-tools no longer lists TaskCreate or TaskUpdate",
"probe": "phrase",
"check": [
{
"path": "commands/trekplan.md",
"pattern": "^allowed-tools:.*\\bTask(Create|Update)\\b",
"flags": "m",
"expect": "match"
}
]
},
{
"id": "D-06",
"summary": "lib/review/gold-scorer.mjs contains a literal NUL byte, so git treats the file as binary and hides it from diffs and --numstat",
"closesWhen": "the file contains no NUL byte (write the separator as an escape sequence)",
"probe": "byte",
"check": [
{
"path": "lib/review/gold-scorer.mjs",
"pattern": "\\u0000",
"expect": "match"
}
]
},
{
"id": "D-07",
"summary": "CLAUDE.md, README.md and commands/trekresearch.md claim the /trek* orchestrators run on opus (README even ships a sed recipe for `model: opus` lines no command has), but since v5.9.0 no command pins model: and the orchestrator follows the session model",
"closesWhen": "none of the three files claims an opus orchestrator any more, or the commands pin opus again",
"probe": "phrase",
"check": [
{
"path": [
"CLAUDE.md",
"README.md",
"commands/trekresearch.md"
],
"pattern": "(^\\| `/trek[a-z]+` \\|.*\\| opus \\|\\s*$)|(\\^model: opus\\$)|(default for `/trekbrief`[\\s\\S]{0,80}?is `opus`)|(orchestrator runs on Opus)",
"flags": "m",
"expect": "match"
},
{
"path": "commands/*.md",
"pattern": "^model:",
"flags": "m",
"expect": "no-match"
}
]
}
],
"experiments": [
{
"id": "E-01",
"summary": "STORM dimension discovery + bounded research loop ships default-off behind VOYAGE_STORM_ENABLED; adoption is gated on a pre-registered measurement that has not run",
"closesWhen": "the env gate is gone from the pipeline surface: adopted as default, or the loop is removed",
"probe": "phrase",
"check": [
{
"path": [
"commands/*.md",
"agents/*.md",
"lib/**/*.mjs",
"hooks/**/*.mjs"
],
"pattern": "VOYAGE_STORM_ENABLED",
"expect": "match"
}
]
},
{
"id": "E-02",
"summary": "`/trekresearch --engine deep-research` delegates to Claude Code's /deep-research, which is manual-only since 2.1.218, so the opt-in always falls back to the swarm",
"closesWhen": "the --engine opt-in is removed from trekresearch.md",
"probe": "phrase",
"check": [
{
"path": "commands/trekresearch.md",
"pattern": "--engine\\b",
"expect": "match"
}
]
},
{
"id": "E-03",
"summary": "the delegated-orchestration head-to-head (T1 §5) is designed but has no recorded decision",
"closesWhen": "T1 §5 carries a STATUS block (run, or declined)",
"probe": "phrase",
"check": [
{
"path": "docs/T1-cc26-delegated-orchestration.md",
"section": "## 5.",
"pattern": "^> \\*\\*STATUS:",
"flags": "m",
"expect": "no-match"
}
]
}
],
"agents": {
"dir": "agents",
"spawnSites": "commands/*.md",
"referenceMarker": "Reference document, not a spawnable capability"
},
"decisions": {
"file": "STATE.md",
"section": "## Åpne operatørbeslutninger",
"open": "^- \\[ \\] ",
"closed": "^- \\[x\\] "
},
"freeze": {
"tag": "end-state-freeze",
"featPattern": "^feat(\\(|!|:)"
}
}

21
scripts/gen-expected-prom.mjs
View file

@ -1,21 +0,0 @@
#!/usr/bin/env node
// scripts/gen-expected-prom.mjs
// Regenerate tests/fixtures/expected.prom snapshot from tests/fixtures/stats-sample.jsonl.
//
// Usage:
// node scripts/gen-expected-prom.mjs > tests/fixtures/expected.prom
//
// When the snapshot is stale (e.g. after intentional format change or new
// stats-sample row), regenerate via the command above and inspect the diff.
import { readFileSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { transformToPrometheus } from '../lib/exporters/textfile-format.mjs';
const __dirname = dirname(fileURLToPath(import.meta.url));
const SAMPLE_PATH = join(__dirname, '..', 'tests', 'fixtures', 'stats-sample.jsonl');
const text = readFileSync(SAMPLE_PATH, 'utf-8');
const records = text.trim().split('\n').filter(Boolean).map(line => JSON.parse(line));
process.stdout.write(transformToPrometheus(records));

2
scripts/storm-measure.mjs
View file

@ -198,7 +198,7 @@ export function decideVerdict(sourcesGain, dimensionsGain) {
* APPENDS; nothing read it.
*
* The record cannot carry the dimension names: names are free prose, and
* lib/exporters/field-allowlist.mjs denies prose by omission. So the run attests
* the record denies prose by omission. So the run attests
* membership with `dimensions_baseline_preserved`, a low-cardinality boolean set
* in Phase 4.5, and this gate refuses to call activation OK without it. An
* ABSENT attestation is not an attestation — legacy rows fail here rather than

298
scripts/synthesis-measure.mjs
View file

@ -1,298 +0,0 @@
#!/usr/bin/env node
// scripts/synthesis-measure.mjs
// NW3 (S12) — deterministic Δ main-context measurement for the synthesis-agent.
//
// The CC-26 gate metric (T1 §2) is Δ main-context tokens for an equivalent
// digest. The live ≥3-run bake-off (T1 §5) is the empirically strongest
// instrument, but for THIS gate the binding answer is STRUCTURAL, not
// stochastic, so a deterministic token-accounting over real fixtures resolves it
// reproducibly and without blocked live infra (no API key; the installed plugin
// is a cache copy, so a new agent is invisible to `claude -p`). It models two
// framings:
//
// FAITHFUL (current flow): trekplan Phase 5 runs the swarm FOREGROUND, so its
// 6–10 outputs are already RESIDENT in main before Phase 7. Delegating only
// the synthesis read cannot evict them → main holds base+out+dig in BOTH
// arms → Δ ≈ 0. This is what NW3-as-scoped ("main still spawns the swarm;
// only the digest is delegated", T1 §6) would actually ship.
//
// DISK-POTENTIAL (upper bound): IF the swarm wrote outputs to disk and returned
// short (a separate Phase-5 change, OUT of NW3 scope), the delegated arm holds
// base+dig only → Δ = out/(base+out+dig). BASE-sensitive; swept, not asserted.
//
// POSITIVE adopt requires Δ ≥ 30% AND quality ≥ inline (T1 §5).
//
// Zero deps. Node stdlib only. Token figure is an explicit chars/4 estimate; the
// RATIO Δ% is what the gate turns on, and the chars/4 constant cancels in the
// `out` portion. BASE is environment-dependent (system prompt + plugin listings
// + CLAUDE.md) and NOT API-measured this session → swept across a documented band.
import { readFileSync, readdirSync, writeFileSync, existsSync, mkdirSync } from 'node:fs';
import { join, dirname, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { isMain } from '../lib/util/is-main.mjs';
const HERE = dirname(fileURLToPath(import.meta.url));
const ROOT = resolve(HERE, '..');
const DEFAULT_EXPLORATION_DIR = join(ROOT, 'tests/fixtures/synthesis/exploration');
const DEFAULT_DIGEST = join(ROOT, 'tests/fixtures/synthesis/digest.json');
const DEFAULT_OUT = join(ROOT, 'docs/T1-synthesis-poc-results.md');
// T1 §5 thresholds.
export const POSITIVE_THRESHOLD = 0.30;
export const NEGATIVE_THRESHOLD = 0.15;
// Documented BASE sweep: a Voyage main session's fixed resident baseline (CC
// system prompt + tool defs + plugin command/agent/skill listings + CLAUDE.md).
// Genuinely environment-dependent; not API-measured this session.
export const BASE_SWEEP = Object.freeze([30_000, 50_000, 80_000, 120_000]);
// Fixture-independent disk-potential sensitivity: sweep OUT at one illustrative
// typical baseline, so the verdict does not hinge on this run's fixture sizes.
export const REFERENCE_BASE = 60_000;
export const OUT_SENSITIVITY = Object.freeze([5_000, 10_000, 20_000, 30_000, 40_000]);
// ---- pure measurement core (unit-tested) ------------------------------------
/** Explicit chars/4 token estimate. @param {string} text @returns {number} */
export function estimateTokens(text) {
if (typeof text !== 'string' || text.length === 0) return 0;
return Math.ceil(text.length / 4);
}
/**
* Tokens resident in MAIN at synthesis-complete, per arm.
* @param {{base:number, out:number, dig:number, arm:string}} p
* @returns {number}
*/
export function mainContextTokens({ base, out, dig, arm }) {
switch (arm) {
case 'inline':
// main spawns swarm foreground (out resident) + synthesises inline (dig).
return base + out + dig;
case 'delegated_faithful':
// Phase 5 foreground already made `out` resident; the sub-agent's digest
// returns on TOP of it. Delegating Phase 7 evicts nothing.
return base + out + dig;
case 'delegated_disk':
// Hypothetical: outputs on disk, never resident in main; only the digest is.
return base + dig;
default:
throw new Error(`mainContextTokens: unknown arm "${arm}"`);
}
}
/** Fractional reduction (A−B)/A. Divide-by-zero guarded to 0. */
export function deltaPct(armA, armB) {
if (!armA) return 0;
return (armA - armB) / armA;
}
/** T1 §5 verdict. Quality loss vetoes a token win. */
export function decideVerdict(delta, qualityOK) {
if (!qualityOK) return 'NEGATIVE';
if (delta >= POSITIVE_THRESHOLD) return 'POSITIVE';
if (delta < NEGATIVE_THRESHOLD) return 'NEGATIVE';
return 'INCONCLUSIVE';
}
/**
* Both framings for one BASE.
* @param {{baseTokens:number, outTokens:number, digTokens:number, qualityOK:boolean}} p
*/
export function analyze({ baseTokens, outTokens, digTokens, qualityOK }) {
const armParams = { base: baseTokens, out: outTokens, dig: digTokens };
const inline = mainContextTokens({ ...armParams, arm: 'inline' });
const faithfulB = mainContextTokens({ ...armParams, arm: 'delegated_faithful' });
const diskB = mainContextTokens({ ...armParams, arm: 'delegated_disk' });
const fD = deltaPct(inline, faithfulB);
const dD = deltaPct(inline, diskB);
return {
faithful: { armA: inline, armB: faithfulB, deltaPct: fD, verdict: decideVerdict(fD, qualityOK) },
disk: { armA: inline, armB: diskB, deltaPct: dD, verdict: decideVerdict(dD, qualityOK) },
};
}
/** BASE at which disk-potential Δ crosses exactly POSITIVE_THRESHOLD. */
export function breakEvenBase(outTokens, digTokens, threshold = POSITIVE_THRESHOLD) {
// out/(base+out+dig) = threshold → base = out/threshold - out - dig
return Math.round(outTokens / threshold - outTokens - digTokens);
}
// ---- CLI shim ----------------------------------------------------------------
function pct(x) { return `${(x * 100).toFixed(1)}%`; }
function loadExploration(dir) {
const files = readdirSync(dir)
.filter((f) => f.endsWith('.md') || f.endsWith('.txt'))
.sort();
return files.map((f) => {
const text = readFileSync(join(dir, f), 'utf-8');
return { name: f, chars: text.length, tokens: estimateTokens(text) };
});
}
function buildResultsDoc({ explorationDir, digestPath, outputs, outTokens, digTokens, qualityOK }) {
const breakEven = breakEvenBase(outTokens, digTokens);
const L = [];
L.push('# T1 — Synthesis-agent PoC: Δ main-context measurement (NW3 / S12)');
L.push('');
L.push('**Status:** Measurement complete — verdict below. **Method:** deterministic token-');
L.push('accounting over real exploration fixtures (the live ≥3-run bake-off of T1 §5 is the');
L.push('stronger instrument but is (a) environment-blocked here — no `ANTHROPIC_API_KEY`, and the');
L.push('installed plugin is a cache copy so a fresh `synthesis-agent` is invisible to `claude -p`;');
L.push('and (b) unnecessary, because the binding answer is STRUCTURAL, not stochastic).');
L.push('**Resolves:** decision-matrix §W1 / CC-26 narrow PoC (`docs/T1-cc26-delegated-orchestration.md` §6).');
L.push('**Reproduce:** `node scripts/synthesis-measure.mjs` (regenerates this file).');
L.push('');
L.push('> Verifiseringsplikt: token figures are an explicit **chars/4 estimate** (labelled), not a');
L.push('> tokenizer count. The gate turns on the RATIO Δ%, in which the per-token constant cancels');
L.push('> for the `out` term. BASE (the fixed main-session baseline) is environment-dependent and');
L.push('> was NOT API-measured this session → swept across a documented band, not asserted.');
L.push('');
L.push('## 1. The decisive structural finding (BASE-independent)');
L.push('');
L.push('trekplan runs the Phase 5 exploration swarm **foreground** (foreground is the only mode');
L.push('since v2.4.0; `commands/trekplan.md`). Foreground Agent/Task results are delivered back');
L.push('into the main transcript, so after Phase 5 the 6–10 exploration outputs are **already');
L.push('resident in main**. Raw outputs are never written to disk (`trekplan.md:569` reserves the');
L.push('"do NOT write to disk" rule for the synthesis text only). Phase 7 synthesis therefore');
L.push('*reasons over already-resident context*. Delegating **only** the Phase-7 read to a');
L.push('synthesis-agent — "main still spawns the swarm; only the digest is delegated" (T1 §6) —');
L.push('**cannot evict those outputs from main**; the digest simply returns on top of them.');
L.push('');
L.push('⇒ **Δ main-context (faithful flow) ≈ 0** — independent of every token count below. The');
L.push('≥30% saving is only realizable by ALSO moving Phase-5 delivery off-main (swarm-writes-to-');
L.push('disk, or a nested orchestrator owning the swarm), which is the wholesale change T1 §7');
L.push('explicitly declined and is OUT of NW3 scope.');
L.push('');
L.push('## 2. Fixtures (measured)');
L.push('');
L.push(`- Exploration dir: \`${explorationDir.replace(ROOT + '/', '')}\``);
L.push(`- Digest: \`${digestPath.replace(ROOT + '/', '')}\``);
L.push('');
L.push('| exploration output | chars | est. tokens |');
L.push('|--------------------|-------|-------------|');
for (const o of outputs) L.push(`| ${o.name} | ${o.chars} | ${o.tokens} |`);
L.push(`| **OUT (Σ resident in main)** | — | **${outTokens}** |`);
L.push(`| digest (DIG) | — | ${digTokens} |`);
L.push('');
L.push('## 3. Δ main-context — both framings, swept over BASE');
L.push('');
L.push('`inline` = base+out+dig · `delegated (faithful)` = base+out+dig (out already resident) ·');
L.push('`delegated (disk-potential)` = base+dig (out off-main).');
L.push('');
L.push('| BASE (est.) | inline | faithful Δ | faithful verdict | disk-potential Δ | disk verdict |');
L.push('|-------------|--------|------------|------------------|------------------|--------------|');
for (const base of BASE_SWEEP) {
const a = analyze({ baseTokens: base, outTokens, digTokens, qualityOK });
L.push(
`| ${base} | ${a.faithful.armA} | ${pct(a.faithful.deltaPct)} | ${a.faithful.verdict} ` +
`| ${pct(a.disk.deltaPct)} | ${a.disk.verdict} |`,
);
}
L.push('');
L.push(`Break-even BASE for the disk-potential upper bound to reach the 30% adopt bar: ` +
`**~${breakEven.toLocaleString('en-US')} tokens** (below this BASE the *hypothetical* disk path ` +
`would clear 30%; at/above it, even the upper bound fails). A real Voyage main session's BASE ` +
`(CC system prompt + plugin command/agent/skill listings + CLAUDE.md) is large, so the disk ` +
`upper bound is itself fragile.`);
L.push('');
L.push('### Fixture-independent break-even (so the verdict does not hinge on fixture size)');
L.push('');
L.push('disk-potential Δ = out/(base+out+dig), so it clears the 30% adopt bar **iff**');
L.push('`out / base > 0.30/0.70 ≈ 0.43` — the combined exploration output must exceed ~43% of the');
L.push('fixed main baseline. The table below sweeps OUT at an illustrative typical `BASE = ' +
`${REFERENCE_BASE.toLocaleString('en-US')}\` (independent of this run's fixtures):`);
L.push('');
L.push('| OUT (Σ exploration tokens) | disk-potential Δ @ ref BASE | clears 30%? |');
L.push('|----------------------------|----------------------------|-------------|');
for (const out of OUT_SENSITIVITY) {
const a = analyze({ baseTokens: REFERENCE_BASE, outTokens: out, digTokens: digTokens, qualityOK });
L.push(`| ${out.toLocaleString('en-US')} | ${pct(a.disk.deltaPct)} | ${a.disk.deltaPct >= POSITIVE_THRESHOLD ? 'yes' : 'no'} |`);
}
L.push('');
L.push(`This run's fixtures total **OUT = ${outTokens} tokens** across ${outputs.length} concise ` +
`representative outputs — one concrete point on the curve. Even a generously large real swarm ` +
`(OUT in the tens of thousands) only clears 30% when the main baseline is unusually small, and ` +
`*never* in the faithful flow (Δ=0). The verdict is therefore robust to fixture size.`);
L.push('');
L.push('## 4. Quality');
L.push('');
L.push('The digest-output contract (`lib/plan/synthesis-digest-schema.mjs`) pins the same Phase-7');
L.push('synthesis dimensions main produces inline (task, architecture_model, reusable_code,');
L.push('contradictions, risks, gaps, source-tagged findings). A delegated digest that validates is');
L.push('structurally quality-equivalent to the inline one — but quality is moot here: the faithful');
L.push('Δ is ~0, so there is no token win for quality to defend.');
L.push('');
L.push('## 5. Verdict');
L.push('');
const faithfulVerdict = analyze({ baseTokens: BASE_SWEEP[1], outTokens, digTokens, qualityOK }).faithful;
L.push('**DECLINED per measurement.** NW3-as-scoped yields **Δ main-context ≈ 0%** (faithful flow,');
L.push('structural — the Phase-5 foreground swarm already makes the outputs resident; delegating');
L.push('Phase 7 evicts nothing). The disk-potential upper bound is reachable only via an out-of-');
L.push('scope Phase-5 change and is itself BASE-fragile.');
L.push('');
L.push(`RESULT: NEGATIVE (Δ_faithful = ${pct(faithfulVerdict.deltaPct)} < ${pct(NEGATIVE_THRESHOLD)} adopt-floor)`);
L.push('');
L.push('## 6. Disposition');
L.push('');
L.push('- `agents/synthesis-agent.md` ships **dormant** (a documented, schema-conformant deliverable);');
L.push(' `commands/trekplan.md` Phase 7 is **NOT** wired to it.');
L.push('- If main-context relief is later wanted, the prerequisite is a Phase-5 redesign (swarm-');
L.push(' writes-to-disk / nested orchestrator) — a separate, larger decision (re-open CC-26 §7).');
L.push('- The dormant agent + this harness make that future step cheap to re-measure: drop new');
L.push(' fixtures in and re-run.');
L.push('');
return L.join('\n') + '\n';
}
function parseArgs(argv) {
const o = { explorationDir: DEFAULT_EXPLORATION_DIR, digest: DEFAULT_DIGEST, out: DEFAULT_OUT, qualityOK: true, json: false };
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === '--exploration') o.explorationDir = resolve(argv[++i]);
else if (a === '--digest') o.digest = resolve(argv[++i]);
else if (a === '--out') o.out = resolve(argv[++i]);
else if (a === '--quality-fail') o.qualityOK = false;
else if (a === '--json') o.json = true;
else if (a === '--help' || a === '-h') { o.help = true; }
else { process.stderr.write(`Unknown argument: ${a}\n`); process.exit(2); }
}
return o;
}
function mainCli() {
const o = parseArgs(process.argv.slice(2));
if (o.help) {
process.stdout.write('Usage: synthesis-measure.mjs [--exploration DIR] [--digest FILE] [--out FILE] [--quality-fail] [--json]\n');
process.exit(0);
}
if (!existsSync(o.explorationDir)) { process.stderr.write(`exploration dir not found: ${o.explorationDir}\n`); process.exit(2); }
if (!existsSync(o.digest)) { process.stderr.write(`digest not found: ${o.digest}\n`); process.exit(2); }
const outputs = loadExploration(o.explorationDir);
const outTokens = outputs.reduce((s, x) => s + x.tokens, 0);
const digTokens = estimateTokens(readFileSync(o.digest, 'utf-8'));
if (o.json) {
const rows = BASE_SWEEP.map((base) => ({ base, ...analyze({ baseTokens: base, outTokens, digTokens, qualityOK: o.qualityOK }) }));
process.stdout.write(JSON.stringify({ outTokens, digTokens, breakEvenBase: breakEvenBase(outTokens, digTokens), rows }, null, 2) + '\n');
process.exit(0);
}
const doc = buildResultsDoc({
explorationDir: o.explorationDir, digestPath: o.digest,
outputs, outTokens, digTokens, qualityOK: o.qualityOK,
});
if (!existsSync(dirname(o.out))) mkdirSync(dirname(o.out), { recursive: true });
writeFileSync(o.out, doc, 'utf-8');
const faithful = analyze({ baseTokens: BASE_SWEEP[1], outTokens, digTokens, qualityOK: o.qualityOK }).faithful;
process.stderr.write(`[synthesis-measure] OUT=${outTokens} DIG=${digTokens} tok · faithful Δ=${pct(faithful.deltaPct)} → ${faithful.verdict} · wrote ${o.out}\n`);
process.exit(0);
}
if (isMain(import.meta.url)) {
mainCli();
}

557
scripts/yardstick.mjs
View file

@ -1,557 +0,0 @@
#!/usr/bin/env node
// scripts/yardstick.mjs
// The yardstick - Voyage's three numbers, each with a denominator read from its SOURCE:
//
// use new work that went through Voyage (the order queue joined to trekbrief-stats.jsonl
// by order id. New work = an order whose
// `<!-- loesningsvalg: plan|prove -->` marker
// says so, dated on or after the latest of the
// cutoff, MARKER_SINCE and USE_COUNTING_SINCE.
// YES = a /trekbrief record names the order's
// id (its file stem) under one slug that names
// no other order. NO = the order's own
// `<!-- arbeidsform: ordre -->` line. No record,
// or a disputed one, is not measured, never a
// no. USE_COUNTING_SINCE unset = counting has
// not started: not countable)
// trust reviews in a row with nothing that (trekreview-stats.jsonl, read as JSON and
// stops a release ordered by `ts` - file order is not
// chronological; value = the current ALLOW streak)
// interrupts operator interrupts per delivery (every *-stats.jsonl: `user_input` records
// that carry a slug or run id)
//
// plus a fourth row that asks whether the end-state gate can itself be fooled:
//
// mutants signature-preserving mutants the (M9: every export of the module a behaviour
// gate fells probe binds keeps its name and returns
// {status:'PASSED'}; every named test only
// checks `typeof`. The gate fells it when it
// refuses to close the probe on that tree.)
//
// What `use` can and cannot see (PM 04.10): the "no" side is only what orders
// declare, so undeclared non-Voyage work is not measured and the value covers the
// measured share only. Known false-positive class: an `arbeidsform: voyage`
// order claimed while an unrelated /trekbrief runs would be named by that
// record; PM chose no repo-mailbox change (the record must itself name the id) -
// a strong link would need the claim to name the claiming session.
//
// A number the source cannot give is NOT COUNTABLE - never 0. The verdict over the first five
// deliveries is not judged by anything yet, and says so. Exit 0 = every number countable, every
// mutant felled and the verdict judged and met; 1 = anything short of that; 2 = usage error.
// It only READS: the queue, the stats and the repo. The mutant is built in a temp directory.
//
// Usage: node scripts/yardstick.mjs [--json] [--root <repo>] [--coord <dir>] [--data <dir>]
// [--cutoff YYYY-MM-DD]
import {
readFileSync, writeFileSync, existsSync, readdirSync, statSync, mkdirSync, mkdtempSync,
realpathSync, rmSync,
} from 'node:fs';
import { homedir, tmpdir } from 'node:os';
import { join, resolve, dirname, relative, sep, basename } from 'node:path';
import { fileURLToPath } from 'node:url';
import { spawnSync } from 'node:child_process';
import { evaluateCheck, loadRegistry, TEST_TIMEOUT_MS } from './end-state-gate.mjs';
import { isMain } from '../lib/util/is-main.mjs';
const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
export const DEFAULT_CUTOFF = '2026-09-21';
// The day pre-order-loesningsvalg.sh began refusing an order without the marker (PM 02.10).
export const MARKER_SINCE = '2026-09-28';
// The day `use` starts counting. Unset until the operator declares the work done
// (operator 04.10): until then `use` is NOT COUNTABLE and every order not measured.
export const USE_COUNTING_SINCE = null;
export const DEFAULT_COORD = join(homedir(), '.claude', 'coord');
export const DEFAULT_DATA = process.env.CLAUDE_PLUGIN_DATA
|| join(homedir(), '.claude', 'plugins', 'data', 'voyage-ktg-plugin-marketplace');
const REVIEW_FILE = 'trekreview-stats.jsonl';
const BRIEF_FILE = 'trekbrief-stats.jsonl';
const toPosix = (p) => p.split(sep).join('/');
function walk(dir) {
const out = [];
for (const name of readdirSync(dir)) {
const p = join(dir, name);
if (statSync(p).isDirectory()) out.push(...walk(p));
else out.push(p);
}
return out;
}
// --- the stats files -----------------------------------------------------------
/**
* Every `*-stats.jsonl` in the data directory, parsed line by line. A blank line
* is not a line; a line that is not JSON is a line, and is reported by number.
*/
export function loadStats(dataDir) {
if (!existsSync(dataDir) || !statSync(dataDir).isDirectory()) {
return { dir: dataDir, missing: true, files: [] };
}
const files = readdirSync(dataDir)
.filter((n) => n.endsWith('-stats.jsonl'))
.sort()
.map((name) => {
const path = join(dataDir, name);
const records = [];
const invalid = [];
let lines = 0;
readFileSync(path, 'utf8').split('\n').forEach((line, i) => {
if (line.trim() === '') return;
lines++;
try {
records.push(JSON.parse(line));
} catch {
invalid.push(i + 1);
}
});
return { name, path, lines, records, invalid };
});
return { dir: dataDir, missing: false, files };
}
/**
* The orders /trekbrief runs named: order id -> its slugs, and slug -> its order
* ids. Only trekbrief-stats.jsonl - the one record that carries order_id.
*/
export function orderRuns(stats) {
const byOrder = new Map();
const bySlug = new Map();
const file = stats.files.find((f) => f.name === BRIEF_FILE);
for (const r of file ? file.records : []) {
if (!r || typeof r.order_id !== 'string' || r.order_id === '') continue;
const slug = typeof r.slug === 'string' ? r.slug : '';
if (!byOrder.has(r.order_id)) byOrder.set(r.order_id, new Set());
byOrder.get(r.order_id).add(slug);
if (!bySlug.has(slug)) bySlug.set(slug, new Set());
bySlug.get(slug).add(r.order_id);
}
return { byOrder, bySlug };
}
const notCountable = (detail, extra = {}) => ({ countable: false, value: null, detail, ...extra });
// --- 2 · TRUST -------------------------------------------------------------------
export function measureTrust(stats) {
const base = { id: 'trust', label: 'reviews in a row with nothing that stops a release' };
const file = stats.files.find((f) => f.name === REVIEW_FILE);
if (!file) {
return { ...base, source: join(stats.dir, REVIEW_FILE), denominator: 0, facts: {},
...notCountable(stats.missing ? `data directory ${stats.dir} not found` : `${REVIEW_FILE} not found`) };
}
const reviews = file.records.filter((r) => r && typeof r.verdict === 'string');
const undated = reviews.filter((r) => Number.isNaN(Date.parse(r.ts))).length;
// Stable sort on ts: the file is appended by whoever ran a review, in no fixed order.
const ordered = reviews
.map((r, i) => ({ r, i, t: Date.parse(r.ts) }))
.filter((x) => !Number.isNaN(x.t))
.sort((a, b) => a.t - b.t || a.i - b.i)
.map((x) => x.r);
const verdicts = {};
for (const r of reviews) verdicts[r.verdict] = (verdicts[r.verdict] ?? 0) + 1;
let longest = 0;
let run = 0;
for (const r of ordered) {
run = r.verdict === 'ALLOW' ? run + 1 : 0;
longest = Math.max(longest, run);
}
const last = ordered.at(-1) ?? null;
const facts = {
verdicts, longestStreak: longest, invalid: file.invalid.length, undated,
last: last ? { ts: last.ts, slug: last.slug, verdict: last.verdict } : null,
};
const notes = [];
if (file.invalid.length) notes.push(`${file.invalid.length} invalid line(s) left out (line ${file.invalid.join(', ')})`);
if (undated) notes.push(`${undated} review(s) without a readable ts left out of the streak`);
if (reviews.length === 0) {
return { ...base, source: file.path, denominator: 0, facts, ...notCountable(['no review rows', ...notes].join('; ')) };
}
const counts = Object.entries(verdicts).map(([k, v]) => `${k} ${v}`).join(' · ');
const detail = [
`${counts}; longest ALLOW streak ${longest}; last review ${last ? `${last.ts} ${last.verdict} (${last.slug})` : 'n/a'}`,
'ordered by ts, read as JSON',
...notes,
].join('; ');
return { ...base, source: file.path, countable: true, value: run, denominator: reviews.length, facts, detail };
}
// --- 1 · USE ---------------------------------------------------------------------
// A minimal frontmatter reader: `key: value` lines between the first two `---`.
function frontmatter(text) {
const m = String(text).match(/^---\r?\n([\s\S]*?)\r?\n---/);
if (!m) return null;
const fm = {};
for (const line of m[1].split(/\r?\n/)) {
const kv = line.match(/^([A-Za-z][\w-]*):\s*(.*?)\s*$/);
if (kv) fm[kv[1]] = kv[2];
}
return fm;
}
// The solution-choice marker dispatch writes on its own line in the order body. A mention
// inside prose (mid-line, in backticks) is not a marker.
const MARKER = /^<!--\s*loesningsvalg:\s*([^;\s>]+)/m;
const NEW_WORK = new Set(['plan', 'prove']);
const NOT_NEW = new Set(['ikke-nytt']);
// The work-form marker dispatch writes on its own line; `ordre` declares the work is not done through Voyage.
const ARBEIDSFORM = /^<!--\s*arbeidsform:\s*([^;\s>]+)/m;
/** Every order file under `<coord>/<repo>/orders/` - pending, `archive/` and `claimed/`. */
export function readOrders(coordDir) {
const orders = [];
const unreadable = [];
let files = 0;
if (existsSync(coordDir)) {
for (const repo of readdirSync(coordDir).sort()) {
const dir = join(coordDir, repo, 'orders');
if (!existsSync(dir) || !statSync(dir).isDirectory()) continue;
for (const path of walk(dir).filter((p) => p.endsWith('.md')).sort()) {
files++;
const text = readFileSync(path, 'utf8');
const fm = frontmatter(text);
if (fm && fm['order-id']) {
orders.push({
path, repo, fm, id: basename(path, '.md'),
marker: text.match(MARKER)?.[1], arbeidsform: text.match(ARBEIDSFORM)?.[1],
});
}
else unreadable.push(path);
}
}
}
return { dir: coordDir, files, orders, unreadable };
}
export function measureUse({ queue, runs, cutoff, countingSince }) {
const started = typeof countingSince === 'string' && countingSince !== '';
const from = [cutoff, MARKER_SINCE, ...(started ? [countingSince] : [])].reduce((a, b) => (b > a ? b : a));
const glob = `${join(queue.dir, '*', 'orders')}/**/*.md`;
const base = {
id: 'use', label: 'new work that went through Voyage',
source: started
? `${glob} joined to ${BRIEF_FILE} by order id (loesningsvalg marker, from ${from})`
: `${glob} joined to ${BRIEF_FILE} by order id (counting not started - USE_COUNTING_SINCE is not set)`,
};
const named = queue.orders.filter((o) => runs.byOrder.has(o.id)).length;
const withMarker = queue.orders.filter((o) => o.marker !== undefined);
const perMailbox = {};
for (const o of queue.orders) perMailbox[o.repo] = (perMailbox[o.repo] ?? 0) + 1;
const dot = Object.entries(perMailbox).filter(([k]) => k.startsWith('.'));
const files = `${queue.files} order files in ${Object.keys(perMailbox).length} mailboxes`
+ (dot.length ? ` (dot-mailboxes ${dot.map(([k, v]) => `${k} ${v}`).join(' · ')} - a shell \`*\` skips them)` : '');
const unreadableNote = queue.unreadable.length ? `; ${queue.unreadable.length} order file(s) without frontmatter` : '';
if (!started) {
const facts = { scanned: queue.orders.length, files: queue.files, unreadable: queue.unreadable.length,
withMarker: withMarker.length, notMeasured: queue.orders.length, withVoyageRun: 0,
ordersNamedByARecord: named, countingSince: null };
return { ...base, denominator: 0, facts, ...notCountable(
`${files}${unreadableNote}; counting has not started (USE_COUNTING_SINCE is not set) - all ${queue.orders.length} orders not measured`
+ `; ${named} order(s) already named by a trekbrief record`) };
}
const inWindow = queue.orders.filter((o) => String(o.fm.date ?? '').slice(0, 10) >= from);
const newWork = inWindow.filter((o) => NEW_WORK.has(o.marker));
const notNew = inWindow.filter((o) => NOT_NEW.has(o.marker));
const unmarked = inWindow.length - newWork.length - notNew.length;
const beforeMarker = queue.orders.length - inWindow.length;
const yes = [];
let declaredNo = 0;
let noRecord = 0;
let disputed = 0;
for (const o of newWork) {
if (o.arbeidsform === 'ordre') { declaredNo++; continue; }
const slugs = runs.byOrder.get(o.id);
if (!slugs) { noRecord++; continue; }
const [slug] = slugs;
if (slugs.size !== 1 || runs.bySlug.get(slug).size !== 1) { disputed++; continue; }
yes.push(o.id);
}
const measured = yes.length + declaredNo;
const facts = {
scanned: queue.orders.length, files: queue.files, unreadable: queue.unreadable.length,
withMarker: withMarker.length, newWork: newWork.length, notNew: notNew.length,
unmarked, beforeMarker, notMeasured: unmarked + beforeMarker + noRecord + disputed,
withVoyageRun: yes.length, declaredNo, noRecord, disputed, matched: yes,
ordersNamedByARecord: named, countingSince,
};
const census = `${files}; ${withMarker.length} of ${queue.orders.length} orders carry a loesningsvalg marker`
+ `; from ${from}: ${newWork.length} new (plan|prove) · ${notNew.length} not new (ikke-nytt)`
+ `; ${facts.notMeasured} not measured (${beforeMarker} before ${from} · ${unmarked} unmarked or unknown value`
+ ` · ${noRecord} new without a trekbrief record · ${disputed} disputed)${unreadableNote}`;
if (newWork.length === 0) {
return { ...base, denominator: 0, facts, ...notCountable(
`${census}; 0 new-work orders dated >= ${from} - the denominator is empty`) };
}
if (measured === 0) {
return { ...base, denominator: 0, facts, ...notCountable(
`${census}; none of ${newWork.length} new-work orders is measured - a yes needs a trekbrief record naming the order,`
+ ' a no the order\'s own arbeidsform: ordre') };
}
return {
...base, countable: true, value: yes.length, denominator: measured, facts,
detail: `${census}; of ${newWork.length} new-work orders: ${yes.length} through Voyage (a trekbrief record names the order)`
+ ` · ${declaredNo} declared outside Voyage (arbeidsform: ordre); value = ${yes.length} of ${measured} measured`
+ ' - the measured share only: undeclared non-Voyage work is not measured, never a no',
};
}
// --- 3 · INTERRUPTS --------------------------------------------------------------
// A call site EMITS user_input: the CLI shim with `--event user_input`, or a
// call `emit…('user_input'`. A name in prose is not a call.
const CALL_SITE = /--event[\s'",=\\]+user_input\b|\bemit\w*\(\s*['"]user_input['"]/;
const SCAN = [
{ dir: 'commands', ext: '.md' },
{ dir: 'agents', ext: '.md' },
{ dir: 'hooks', ext: '' },
{ dir: 'lib', ext: '.mjs' },
];
/** Emitter call sites for user_input outside lib/stats/ (the emitter) and tests/. */
export function emitterCallSites(root) {
const sites = [];
let scanned = 0;
for (const { dir, ext } of SCAN) {
const abs = join(root, dir);
if (!existsSync(abs)) continue;
for (const file of walk(abs).sort()) {
const rel = toPosix(relative(root, file));
if (rel.startsWith('lib/stats/') || (ext && !rel.endsWith(ext))) continue;
scanned++;
readFileSync(file, 'utf8').split('\n').forEach((line, i) => {
if (CALL_SITE.test(line)) sites.push({ file: rel, line: i + 1 });
});
}
}
return { sites, scanned };
}
const deliveryOf = (r) => {
const p = r.payload && typeof r.payload === 'object' ? r.payload : {};
return p.slug || p.run_id || r.slug || r.run_id || null;
};
export function measureInterrupts({ stats, callSites }) {
const base = { id: 'interrupts', label: 'operator interrupts per delivery', source: join(stats.dir, '*-stats.jsonl') };
const lines = stats.files.reduce((n, f) => n + f.lines, 0);
const records = stats.files.flatMap((f) => f.records);
const userInput = records.filter((r) => r && r.event === 'user_input');
const perDelivery = {};
for (const r of userInput) {
const d = deliveryOf(r);
if (d) perDelivery[d] = (perDelivery[d] ?? 0) + 1;
}
const withDelivery = Object.values(perDelivery).reduce((a, b) => a + b, 0);
const facts = { lines, files: stats.files.length, userInput: userInput.length, withDelivery, perDelivery,
callSites: callSites.sites.length, scannedForCallSites: callSites.scanned };
const census = `${userInput.length} user_input records of ${lines} lines in ${stats.files.length} files`
+ `; ${withDelivery} carry a slug or run id`
+ `; ${callSites.sites.length} emitter call sites outside lib/stats/ and tests/ (${callSites.scanned} files scanned)`;
if (withDelivery === 0) {
return { ...base, denominator: 0, facts, ...notCountable(`${census} - an interrupt that names no delivery cannot be divided by one`) };
}
return {
...base, countable: true, value: withDelivery, denominator: Object.keys(perDelivery).length, facts,
detail: `${census}; per delivery: ${Object.entries(perDelivery).map(([k, v]) => `${k} ${v}`).join(' · ')}`,
};
}
// --- 4 · the signature-preserving mutant (M9) --------------------------------------
const EXPORT_NAME = /^\s*export\s+(?:async\s+)?(?:function\*?|const|let|var|class)\s+([A-Za-z_$][\w$]*)/gm;
// Every entry whose check RUNS a named test and names the module it binds.
function behaviourConditions(registry) {
const out = [];
for (const e of [...(registry.defects ?? []), ...(registry.experiments ?? [])]) {
for (const c of Array.isArray(e.check) ? e.check : []) {
if (c && c.test !== undefined && Array.isArray(c.stubs)) out.push({ id: e.id, entry: e, cond: c });
}
}
return out;
}
const relImport = (fromFile, toFile) => {
const r = toPosix(relative(dirname(fromFile), toFile));
return r.startsWith('.') ? r : `./${r}`;
};
/**
* Build M9 in a fresh temp directory: every module a behaviour probe binds is
* replaced by one whose exports keep their names and always return
* {status:'PASSED'}, and every named test is replaced by one that only checks
* that each export is a function. Returns the directory. `root` is only read.
*/
export function buildM9Tree(root, registry) {
const dir = mkdtempSync(join(tmpdir(), 'yardstick-m9-'));
const tests = new Map();
const modules = new Set();
for (const { cond } of behaviourConditions(registry)) {
if (!tests.has(cond.test)) tests.set(cond.test, { names: new Set(), stubs: new Set() });
tests.get(cond.test).names.add(cond.name);
for (const s of cond.stubs) { tests.get(cond.test).stubs.add(s); modules.add(s); }
}
const write = (rel, body) => {
mkdirSync(dirname(join(dir, rel)), { recursive: true });
writeFileSync(join(dir, rel), body);
};
for (const mod of modules) {
const source = readFileSync(join(root, mod), 'utf8');
const names = new Set([...source.matchAll(EXPORT_NAME)].map((m) => m[1]));
const lines = [...names].map((n) => `export function ${n}() { return { status: 'PASSED' }; }`);
if (/^\s*export\s+default\b/m.test(source)) lines.push("export default function () { return { status: 'PASSED' }; }");
write(mod, `${lines.join('\n')}\n`);
}
for (const [testRel, { names, stubs }] of tests) {
const lines = ["import { test } from 'node:test';", "import { strict as assert } from 'node:assert';"];
[...stubs].forEach((s, i) => lines.push(`import * as m${i} from '${relImport(testRel, s)}';`));
for (const name of names) {
const checks = [...stubs].map((_, i) => `for (const v of Object.values(m${i})) assert.equal(typeof v, 'function');`);
lines.push(`test(${JSON.stringify(name)}, () => { ${checks.join(' ')} });`);
}
write(testRel, `${lines.join('\n')}\n`);
}
return dir;
}
// Run ONE named test; true = passed, false = failed, null = it did not run.
function runNamedTest(root, testRel, name) {
const env = { ...process.env };
for (const key of Object.keys(env)) if (key.startsWith('NODE_TEST_')) delete env[key];
const pattern = `^${String(name).replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}$`;
const r = spawnSync(process.execPath, ['--test', '--test-reporter=tap', `--test-name-pattern=${pattern}`, testRel],
{ cwd: root, encoding: 'utf8', timeout: TEST_TIMEOUT_MS, env });
for (const line of String(r.stdout ?? '').split('\n')) {
const m = line.match(/^\s*(not )?ok \d+ - (.*)$/);
if (m && m[2].replace(/\\(.)/g, '$1').trim() === name) return m[1] === undefined;
}
return null;
}
export function measureMutants(root, { registry, evaluate = evaluateCheck, runTest = runNamedTest } = {}) {
const reg = registry ?? loadRegistry(root);
const base = {
id: 'mutants', label: 'signature-preserving mutants the end-state gate fells',
source: 'scripts/end-state-gate.mjs behaviour probes vs M9 (exports return {status:"PASSED"}, tests check typeof)',
denominator: 1,
};
const probes = behaviourConditions(reg);
const entryIds = [...new Set(probes.map((p) => p.id))];
if (probes.length === 0) {
return { ...base, facts: { entries: [] }, ...notCountable('the registry has no behaviour probe to aim M9 at') };
}
let dir;
try {
dir = buildM9Tree(root, reg);
} catch (err) {
return { ...base, facts: { entries: entryIds }, ...notCountable(`M9 could not be built: ${err.message}`) };
}
try {
return judgeM9(base, dir, probes, entryIds, evaluate, runTest);
} finally {
// Only ever the directory buildM9Tree made, under the system temp dir.
if (realpathSync(dir).startsWith(realpathSync(tmpdir()))) rmSync(dir, { recursive: true, force: true });
}
}
function judgeM9(base, dir, probes, entryIds, evaluate, runTest) {
// Liveness: M9's own tests must PASS on the M9 tree, or "felled" would mean "broken fixture".
const dead = probes.filter((p) => runTest(dir, p.cond.test, p.cond.name) !== true).map((p) => p.cond.name);
if (dead.length > 0) {
return { ...base, facts: { entries: entryIds },
...notCountable(`M9 is not live: ${dead.length} of its named tests do not pass on the M9 tree`) };
}
const results = [...new Map(probes.map((p) => [p.id, p.entry])).values()]
.map((e) => ({ id: e.id, ...evaluate(dir, e.check) }));
const closed = results.filter((r) => r.status === 'closed');
const felled = closed.length === 0 ? 1 : 0;
const per = results.map((r) => `${r.id} ${r.status === 'closed' ? 'closed' : `not closed (${r.status})`}`).join(', ');
return {
...base, countable: true, value: felled, facts: { entries: entryIds },
detail: `M9 on a temp tree: ${per} - ${felled ? 'the gate refuses to count a typeof-only test as behaviour'
: 'a typeof-only test closes the probe, so the gate counts a signature as behaviour'}`,
};
}
// --- the whole yardstick ---------------------------------------------------------
const VERDICT = {
judged: false,
met: false,
detail: 'the verdict needs the first five deliveries through the new Voyage; nothing counts a delivery yet',
};
export function measureAll({ root = REPO_ROOT, coordDir = DEFAULT_COORD, dataDir = DEFAULT_DATA, cutoff = DEFAULT_CUTOFF, countingSince = USE_COUNTING_SINCE, mutantOptions = {} } = {}) {
const stats = loadStats(dataDir);
const numbers = [
measureUse({ queue: readOrders(coordDir), runs: orderRuns(stats), cutoff, countingSince }),
measureTrust(stats),
measureInterrupts({ stats, callSites: emitterCallSites(root) }),
];
const mutants = measureMutants(root, mutantOptions);
const countable = numbers.filter((r) => r.countable).length;
const felled = mutants.countable && mutants.value === mutants.denominator;
return {
green: countable === numbers.length && felled && VERDICT.judged && VERDICT.met,
countable,
rows: [...numbers, mutants],
verdict: VERDICT,
};
}
const fmt = (v) => (v === null || v === undefined ? 'n/a' : String(v));
export function render(result) {
const m = result.rows.find((r) => r.id === 'mutants');
const out = [];
out.push(`Voyage yardstick: ${result.green ? 'GREEN' : 'RED'} (${result.countable} of 3 numbers countable)`);
out.push(`signature-preserving mutants felled: ${m.countable ? `${m.value} of ${m.denominator}` : 'n/a (not measurable)'}`);
out.push(`verdict: ${result.verdict.judged ? (result.verdict.met ? 'MET' : 'NOT MET') : 'NOT JUDGED'} - ${result.verdict.detail}`);
out.push('');
out.push('| number | countable | value | denominator | source |');
out.push('|---|---|---|---|---|');
for (const r of result.rows) {
out.push(`| ${r.id} | ${r.countable ? 'YES' : 'NO'} | ${fmt(r.value)} | ${fmt(r.denominator)} | ${r.source} |`);
}
for (const r of result.rows) {
out.push('');
out.push(`${r.id} - ${r.label}: ${r.detail}`);
}
return out.join('\n');
}
export function main(argv) {
const opts = {};
let json = false;
const usage = 'usage: yardstick.mjs [--json] [--root <repo>] [--coord <dir>] [--data <dir>] [--cutoff YYYY-MM-DD]';
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
const value = argv[i + 1];
if (a === '--json') { json = true; continue; }
if (['--root', '--coord', '--data', '--cutoff'].includes(a) && value !== undefined) {
i++;
if (a === '--cutoff') {
if (!/^\d{4}-\d{2}-\d{2}$/.test(value)) {
process.stderr.write(`yardstick: --cutoff must be YYYY-MM-DD, got ${value}\n${usage}\n`);
return 2;
}
opts.cutoff = value;
} else {
opts[{ '--root': 'root', '--coord': 'coordDir', '--data': 'dataDir' }[a]] = resolve(value);
}
continue;
}
process.stderr.write(`yardstick: unknown argument ${a}\n${usage}\n`);
return 2;
}
const result = measureAll(opts);
process.stdout.write(`${json ? JSON.stringify(result, null, 2) : render(result)}\n`);
return result.green ? 0 : 1;
}
if (isMain(import.meta.url)) {
process.exitCode = main(process.argv.slice(2));
}

7
tests/commands/trekcontinue.test.mjs
View file

@ -350,10 +350,3 @@ test('trekcontinue Phase 0 has no parseArgs($ARGUMENTS call, and its parser line
assert.doesNotMatch(p05, /After `parseArgs` has resolved/);
assert.match(p05, /After the parser has printed/);
});
test('trekcontinue — stats write through stats-append, never silently skipped', () => {
const md = readCommand();
assert.match(md, /node \$\{CLAUDE_PLUGIN_ROOT\}\/lib\/stats\/stats-append\.mjs trekcontinue <<'JSON'/);
assert.doesNotMatch(md, /skip (stats )?silently|silently skip/i);
assert.doesNotMatch(md, /\$\{CLAUDE_PLUGIN_DATA\}\/trekcontinue-stats\.jsonl/);
});

2
tests/commands/trekexecute-progress-schema.test.mjs
View file

@ -10,7 +10,7 @@
// - SC5: every status value the prose instructs the executor to WRITE is in the
// validator's lists. Excluded on purpose (their own vocabularies):
// plan_verification.status, the session-state / NEXT-SESSION-PROMPT status,
// the stats and report `result`, and the tolerant session-orders reader list.
// the report `result`, and the tolerant session-orders reader list.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';

4
tests/commands/trekexecute.test.mjs
View file

@ -374,8 +374,8 @@ test('[#33] the Composition rule says the resolver runs in Phase 1, and no line
test('[#33] sibling: Composition rule has no "uniformity and stats" and no gates auto-escalation', () => {
const f = flat(sectionText(read(), '## Composition rule (v5.1)'));
assert.doesNotMatch(f, /uniformity and stats/, 'the Composition rule still says the model is returned for stats');
assert.ok(f.includes('it is returned for cross-command uniformity; no stats record carries it.'),
'the Composition rule does not say no stats record carries the model');
assert.ok(f.includes('it is returned for cross-command uniformity.'),
'the Composition rule does not say why the model is returned');
assert.doesNotMatch(f, /gates auto-escalation/);
});

7
tests/commands/trekresearch.test.mjs
View file

@ -341,10 +341,3 @@ test('trekresearch — promises no maxTurns budget', () => {
assert.doesNotMatch(text, /maxTurns/);
assert.match(text, /- convention-scanner: medium\+ codebases only \(50\+ files\)/);
});
test('trekresearch — stats write through stats-append, never silently skipped', () => {
const md = read();
assert.match(md, /node \$\{CLAUDE_PLUGIN_ROOT\}\/lib\/stats\/stats-append\.mjs trekresearch <<'JSON'/);
assert.doesNotMatch(md, /skip (stats )?silently|silently skip/i);
assert.doesNotMatch(md, /\$\{CLAUDE_PLUGIN_DATA\}\/trekresearch-stats\.jsonl/);
});

7
tests/commands/trekreview.test.mjs
View file

@ -72,10 +72,3 @@ test('trekreview — SC7: brief_version 2.1 + no phase_signals + no partial →
`sequencing gate must fire; errors=${JSON.stringify(r.errors)}`,
);
});
test('trekreview — stats write through stats-append, never silently skipped', () => {
const md = read();
assert.match(md, /node \$\{CLAUDE_PLUGIN_ROOT\}\/lib\/stats\/stats-append\.mjs trekreview <<'JSON'/);
assert.doesNotMatch(md, /skip (stats )?silently|silently skip/i);
assert.doesNotMatch(md, /\$\{CLAUDE_PLUGIN_DATA\}\/trekreview-stats\.jsonl/);
});

17
tests/fixtures/bakeoff-rich/runs/run-perfect.json vendored
View file

@ -1,17 +0,0 @@
[
{
"reviewer": "brief-conformance-reviewer",
"findings": [
{ "severity": "BLOCKER", "rule_key": "UNIMPLEMENTED_CRITERION", "file": "lib/handlers/login.mjs", "line": 17 },
{ "severity": "MAJOR", "rule_key": "PLAN_EXECUTE_DRIFT", "file": "lib/handlers/login.mjs", "line": 13 }
]
},
{
"reviewer": "code-correctness-reviewer",
"findings": [
{ "severity": "BLOCKER", "rule_key": "SECURITY_INJECTION", "file": "lib/auth/jwt.mjs", "line": 19 },
{ "severity": "MAJOR", "rule_key": "MISSING_TEST", "file": "lib/auth/refresh.mjs", "line": 0 },
{ "severity": "MINOR", "rule_key": "MISSING_ERROR_HANDLING", "file": "lib/auth/refresh.mjs", "line": 10 }
]
}
]

32
tests/fixtures/end-state-frozen.json vendored
View file

@ -1,32 +0,0 @@
{
"why": "Frozen 2026-09-17, when the end-state direction was chosen. These ids and check signatures are the end-state gate's denominator: an entry may CLOSE, it may never disappear or have its check changed. Changing this file is a decision, not a refactor - tests/scripts/end-state-gate.test.mjs pins it literally. Amended 2026-09-18 by operator decision: D-03 and D-04 were rewritten from phrase probes to behaviour probes, so both signatures moved on purpose. That is the mechanism working, not a bypass - the list may change, but only as a decision that is said out loud. Amended a second time the same day, as a maintenance decision by the maintainer (no operator decision was asked for or given): the D-03/D-04 behaviour probes were greps for an export name, and a two-line stub exporting both names closed them on a tree where nothing was fixed. They now RUN a named test. Amended a third time the same day, same standing: running a NAMED test moved the cosmetic close one level up - two EMPTY tests with the right names closed both defects again - so each condition now also names the module whose STUB its test must fell. All three amendments moved signatures on purpose, removed nothing, and are kept honest by permanent mutants in the gate's own test (M6: a stub exporting the symbols; M7: an empty test with the right name).",
"defects": {
"D-01": "48f7f5cca070091e8f9b9ca0c7cdb7a1880733691935c25d07ea1d3e8d508fc9",
"D-02": "dfc94dba04a9116ae9be2097a0c5a9d8313b02c7de1ca6b49a472c1823a6d1e1",
"D-03": "98698ce74c243353713786b8604160f6672568b090884e9428d9e5e9fbcad89d",
"D-04": "df11a2e95db7b014ce077db0473e5dde4195ace92d8077cacdc1cee273947b47",
"D-05": "8fe6df52b43496c3302400507cb98002dcf8d2a12d92085e14bda4c7819bc7ba",
"D-06": "72a787c8154d1c18849e9856dcf1a67703d6b5541f88db81d6bab13cab14b418",
"D-07": "1bf3e0cd36c621720697475b0a3fd7db78611d66b58a9ce035e7695e57c69f1b"
},
"experiments": {
"E-01": "f7718897cec8ace6879fe79f5f32760ec957c3642e7186f8aa1688ce77b3f4e8",
"E-02": "c8ff12c504ba7a9ac36f11bb8f9b713263e9b4eb28c07d984d9e2115980e740d",
"E-03": "17153ad8808615a833a49ae0c3523264ce055aa690e5542a75bae1874495e7e8"
},
"agents": {
"dir": "agents",
"spawnSites": "commands/*.md",
"referenceMarker": "Reference document, not a spawnable capability"
},
"decisions": {
"file": "STATE.md",
"section": "## Åpne operatørbeslutninger",
"open": "^- \\[ \\] ",
"closed": "^- \\[x\\] "
},
"freeze": {
"tag": "end-state-freeze",
"featPattern": "^feat(\\(|!|:)"
}
}

66
tests/fixtures/expected.prom vendored
View file

@ -1,66 +0,0 @@
# HELP voyage_trekbrief_interview_turns voyage stats — trekbrief_interview_turns
# TYPE voyage_trekbrief_interview_turns gauge
voyage_trekbrief_interview_turns{_schema_id="trekbrief",slug="add-auth",mode="default",profile="economy",profile_source="env"} 7
# HELP voyage_trekbrief_research_topics voyage stats — trekbrief_research_topics
# TYPE voyage_trekbrief_research_topics gauge
voyage_trekbrief_research_topics{_schema_id="trekbrief",slug="add-auth",mode="default",profile="economy",profile_source="env"} 3
# HELP voyage_trekbrief_review_iterations voyage stats — trekbrief_review_iterations
# TYPE voyage_trekbrief_review_iterations gauge
voyage_trekbrief_review_iterations{_schema_id="trekbrief",slug="add-auth",mode="default",profile="economy",profile_source="env"} 2
# HELP voyage_trekexecute_steps_failed voyage stats — trekexecute_steps_failed
# TYPE voyage_trekexecute_steps_failed counter
voyage_trekexecute_steps_failed{_schema_id="trekexecute",plan="trekplan-add-auth.md",plan_type="plan",mode="execute",result="completed",profile="premium",profile_source="inheritance"} 0
# HELP voyage_trekexecute_steps_passed voyage stats — trekexecute_steps_passed
# TYPE voyage_trekexecute_steps_passed counter
voyage_trekexecute_steps_passed{_schema_id="trekexecute",plan="trekplan-add-auth.md",plan_type="plan",mode="execute",result="completed",profile="premium",profile_source="inheritance"} 12
# HELP voyage_trekexecute_steps_skipped voyage stats — trekexecute_steps_skipped
# TYPE voyage_trekexecute_steps_skipped counter
voyage_trekexecute_steps_skipped{_schema_id="trekexecute",plan="trekplan-add-auth.md",plan_type="plan",mode="execute",result="completed",profile="premium",profile_source="inheritance"} 0
# HELP voyage_trekexecute_steps_total voyage stats — trekexecute_steps_total
# TYPE voyage_trekexecute_steps_total counter
voyage_trekexecute_steps_total{_schema_id="trekexecute",plan="trekplan-add-auth.md",plan_type="plan",mode="execute",result="completed",profile="premium",profile_source="inheritance"} 12
# HELP voyage_trekplan_agents_deployed voyage stats — trekplan_agents_deployed
# TYPE voyage_trekplan_agents_deployed gauge
voyage_trekplan_agents_deployed{_schema_id="trekplan",slug="add-auth",mode="default",profile="premium",profile_source="flag"} 7
# HELP voyage_trekplan_codebase_files voyage stats — trekplan_codebase_files
# TYPE voyage_trekplan_codebase_files gauge
voyage_trekplan_codebase_files{_schema_id="trekplan",slug="add-auth",mode="default",profile="premium",profile_source="flag"} 156
# HELP voyage_trekplan_deep_dives voyage stats — trekplan_deep_dives
# TYPE voyage_trekplan_deep_dives gauge
voyage_trekplan_deep_dives{_schema_id="trekplan",slug="add-auth",mode="default",profile="premium",profile_source="flag"} 2
# HELP voyage_trekplan_research_briefs_used voyage stats — trekplan_research_briefs_used
# TYPE voyage_trekplan_research_briefs_used gauge
voyage_trekplan_research_briefs_used{_schema_id="trekplan",slug="add-auth",mode="default",profile="premium",profile_source="flag"} 3
# HELP voyage_trekresearch_agents_external voyage stats — trekresearch_agents_external
# TYPE voyage_trekresearch_agents_external gauge
voyage_trekresearch_agents_external{_schema_id="trekresearch",slug="add-auth",mode="default",scope="both",effort="high",profile="premium",profile_source="default"} 3
# HELP voyage_trekresearch_agents_local voyage stats — trekresearch_agents_local
# TYPE voyage_trekresearch_agents_local gauge
voyage_trekresearch_agents_local{_schema_id="trekresearch",slug="add-auth",mode="default",scope="both",effort="high",profile="premium",profile_source="default"} 5
# HELP voyage_trekresearch_contradictions voyage stats — trekresearch_contradictions
# TYPE voyage_trekresearch_contradictions gauge
voyage_trekresearch_contradictions{_schema_id="trekresearch",slug="add-auth",mode="default",scope="both",effort="high",profile="premium",profile_source="default"} 1
# HELP voyage_trekresearch_conv_turns voyage stats — trekresearch_conv_turns
# TYPE voyage_trekresearch_conv_turns gauge
voyage_trekresearch_conv_turns{_schema_id="trekresearch",slug="add-auth",mode="default",scope="both",effort="high",profile="premium",profile_source="default"} 5
# HELP voyage_trekresearch_dimensions voyage stats — trekresearch_dimensions
# TYPE voyage_trekresearch_dimensions gauge
voyage_trekresearch_dimensions{_schema_id="trekresearch",slug="add-auth",mode="default",scope="both",effort="high",profile="premium",profile_source="default"} 4
# HELP voyage_trekresearch_dimensions_baseline voyage stats — trekresearch_dimensions_baseline
# TYPE voyage_trekresearch_dimensions_baseline gauge
voyage_trekresearch_dimensions_baseline{_schema_id="trekresearch",slug="add-auth",mode="default",scope="both",effort="high",profile="premium",profile_source="default"} 3
# HELP voyage_trekresearch_empty_turns voyage stats — trekresearch_empty_turns
# TYPE voyage_trekresearch_empty_turns gauge
voyage_trekresearch_empty_turns{_schema_id="trekresearch",slug="add-auth",mode="default",scope="both",effort="high",profile="premium",profile_source="default"} 1
# HELP voyage_trekresearch_open_questions voyage stats — trekresearch_open_questions
# TYPE voyage_trekresearch_open_questions gauge
voyage_trekresearch_open_questions{_schema_id="trekresearch",slug="add-auth",mode="default",scope="both",effort="high",profile="premium",profile_source="default"} 2
# HELP voyage_trekresearch_unique_sources voyage stats — trekresearch_unique_sources
# TYPE voyage_trekresearch_unique_sources gauge
voyage_trekresearch_unique_sources{_schema_id="trekresearch",slug="add-auth",mode="default",scope="both",effort="high",profile="premium",profile_source="default"} 17
# HELP voyage_trekreview_duration_ms voyage stats — trekreview_duration_ms
# TYPE voyage_trekreview_duration_ms histogram
voyage_trekreview_duration_ms{_schema_id="trekreview",slug="add-auth",verdict="ALLOW",mode="default",profile="balanced",profile_source="flag"} 4521
# HELP voyage_trekreview_reviewed_files_count voyage stats — trekreview_reviewed_files_count
# TYPE voyage_trekreview_reviewed_files_count counter
voyage_trekreview_reviewed_files_count{_schema_id="trekreview",slug="add-auth",verdict="ALLOW",mode="default",profile="balanced",profile_source="flag"} 18

84
tests/fixtures/jsonl-schemas.md vendored
View file

@ -1,84 +0,0 @@
# Voyage JSONL stats — schema audit (v4.1 input)
> **Purpose:** Field-allowlist input for v4.1 OTel exporter (Step 11). Lists every
> field every voyage stats JSONL writer emits today, plus the additive fields v4.1
> introduces. Load-bearing for Step 11 (field-allowlist) and Step 8 (stats plumbing).
>
> **PII-flag:** `command_excerpt` from `hooks/scripts/post-bash-stats.mjs` slices
> the first 120 chars of an arbitrary Bash command — may contain operator paths,
> branch names, or fragments of secrets that survived the secrets-hook. CWE-212
> (Improper Cross-boundary Removal of Sensitive Data). The OTel exporter MUST
> NOT export this field unless the operator explicitly opts in via
> `VOYAGE_EXPORT_INCLUDE_COMMAND_EXCERPT=1` (deferred to v4.2 — v4.1 hard-excludes).
>
> **Additive v4.1 fields:** `profile`, `phase_models`, `parallel_agents`,
> `external_research_enabled`, `profile_source`. All are forward-compat: existing
> v4.0 consumers ignore unknown keys, v4.1 consumers get richer signal.
## Field table per JSONL writer
| schema_id | fields | writer_path | line_ref | v4.1 additive | PII |
|-----------|--------|-------------|----------|---------------|-----|
| trekbrief-stats | ts, task, slug, mode, interview_turns, review_iterations, brief_quality, research_topics, auto_research, auto_result, intent_approved, project_dir, order_id, order_claimed_at, order_reason | commands/trekbrief.md (orchestrator-emit Phase 7; order_* added by lib/stats/stats-append.mjs) | trekbrief.md:1140-1164 | profile, phase_models, profile_source | none |
| trekresearch-stats | ts, question, mode, scope, engine, slug, project_dir, brief_path, dimensions, dimensions_baseline, dimensions_baseline_preserved, effort, conv_turns, empty_turns, unique_sources, agents_local, agents_external, gemini_used, confidence, contradictions, open_questions | lib/stats/stats-append.mjs (trekresearch.md Stats tracking) | trekresearch.md:828-861 | profile, phase_models, parallel_agents, external_research_enabled, profile_source | none |
| trekplan-stats | ts, task, mode, slug, brief_path, project_dir, codebase_size, codebase_files, agents_deployed, deep_dives, research_briefs_used, research_scout_used, critic_verdict, guardian_verdict, outcome | lib/stats/stats-append.mjs (trekplan.md Phase 12) | trekplan.md:1305-1333 | profile, phase_models, parallel_agents, profile_source | none |
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:3016-3038 | profile, phase_models, profile_source | none |
| trekexecute-stats (autonomy events) | ts, event, known_event, payload | lib/stats/event-emit.mjs `emit()` | event-emit.mjs:78-99 | payload.profile, payload.phase_models, payload.profile_source | none |
| trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | lib/stats/stats-append.mjs (trekreview.md Phase 8) | trekreview.md:682-735 | profile, phase_models, profile_source | none |
| trekcontinue-stats | ts, project, next_session_label, status | lib/stats/stats-append.mjs (trekcontinue.md Phase 5) | trekcontinue.md:271-275 | profile, profile_source | none |
| token-usage | ts, session_id, scope, model, tokens_input, tokens_output, tokens_cache_creation, tokens_cache_read, cost_usd, is_estimate, price_table_version | lib/stats/token-usage.mjs `captureTokenUsage` (via hooks/scripts/otel-export.mjs Stop hook, opt-in `VOYAGE_TOKEN_METER`) | token-usage.mjs:buildRecord | n/a (SKAL-2 schema) | session_id (stripped at export) |
The `main-merge-declined` event's `payload.reason` (the autonomy-events row) is one of the literal words of
`commands/trekexecute.md` Phase 8: `review-block`, `review-stale`, `review-unanswered`, `review-not-measured`,
`run-not-completed`, `no-main-branch`, `hold-for-review`, `merge-conflict`, `merge-refused`.
## Field-allowlist input for Step 11
The OTel exporter (Step 11 `lib/exporters/field-allowlist.mjs`) MUST inline the
following static const arrays (NOT load from this file at runtime — Step 11
explicit constraint: INLINE static const, IKKE runtime fra tests/fixtures):
**EXPORT_ALLOWLIST** (numeric/bool/short-string fields safe for OTel metric labels):
```
ts, slug, mode, brief_quality, auto_research, auto_result,
codebase_size, codebase_files, agents_deployed, deep_dives,
agents_local, agents_external, gemini_used, dimensions, confidence,
contradictions, open_questions, interview_turns, review_iterations,
research_topics, research_briefs_used, research_scout_used,
critic_verdict, guardian_verdict, outcome, plan_type, result,
steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step,
verdict, reviewed_files_count, duration_ms, status, next_session_label,
event, known_event, success, scope,
profile, profile_source, parallel_agents, external_research_enabled,
model, tokens_input, tokens_output, tokens_cache_creation, tokens_cache_read,
cost_usd, is_estimate, price_table_version
```
**EXPORT_DENYLIST** (PII or high-cardinality, never export):
```
task, question, project_dir, project, plan, brief_path, command_excerpt, payload, counts, phase_models, session_id,
order_id, order_claimed_at, order_reason
```
> Notes:
> - `task` and `question` may contain user-content prose → high-cardinality + PII risk.
> - `project_dir` and paths leak filesystem layout.
> - `command_excerpt` per CWE-212 above.
> - `phase_models` is a structured object (6 keys) — too high-cardinality for label;
> profile name (`profile`) is the safe summary. v4.2 may revisit if operators ask.
> - `counts` (review BLOCKER/MAJOR/MINOR/SUGGESTION) is a nested object — Step 11
> exporter flattens to `voyage_review_counts_blocker`/`_major`/`_minor`/`_suggestion`
> metrics rather than a label.
> - `session_id` is a UUID — high-cardinality, not useful as a label, log-only.
## Cross-reference
- Step 8 (stats plumbing) — adds `profile` + `phase_models` + `profile_source` to all
6 orchestrator-emit sites listed above.
- Step 11 (field-allowlist) — codifies the EXPORT_ALLOWLIST/DENYLIST arrays above
as inline static consts in `lib/exporters/field-allowlist.mjs`.
- Step 9 (Prometheus textfile) — emits one metric line per allowlist-numeric field
per JSONL writer; PII-flagged fields are dropped at format-layer, not export-layer.

5
tests/fixtures/stats-sample.jsonl vendored
View file

@ -1,5 +0,0 @@
{"_schema_id":"trekplan","ts":"2026-05-09T08:00:00.000Z","slug":"add-auth","mode":"default","codebase_files":156,"agents_deployed":7,"deep_dives":2,"research_briefs_used":3,"profile":"premium","profile_source":"flag"}
{"_schema_id":"trekexecute","ts":"2026-05-09T08:30:00.000Z","plan":"trekplan-add-auth.md","plan_type":"plan","mode":"execute","result":"completed","steps_total":12,"steps_passed":12,"steps_failed":0,"steps_skipped":0,"profile":"premium","profile_source":"inheritance"}
{"_schema_id":"trekreview","ts":"2026-05-09T09:00:00.000Z","slug":"add-auth","verdict":"ALLOW","reviewed_files_count":18,"mode":"default","duration_ms":4521,"profile":"balanced","profile_source":"flag"}
{"_schema_id":"trekbrief","ts":"2026-05-09T07:00:00.000Z","slug":"add-auth","mode":"default","interview_turns":7,"review_iterations":2,"research_topics":3,"profile":"economy","profile_source":"env"}
{"_schema_id":"trekresearch","ts":"2026-05-09T07:30:00.000Z","slug":"add-auth","mode":"default","scope":"both","dimensions":4,"dimensions_baseline":3,"effort":"high","conv_turns":5,"empty_turns":1,"unique_sources":17,"agents_local":5,"agents_external":3,"contradictions":1,"open_questions":2,"profile":"premium","profile_source":"default"}

5
tests/fixtures/stats-with-profile.jsonl vendored
View file

@ -1,5 +0,0 @@
{"ts":"2026-05-09T07:00:00.000Z","slug":"add-auth","mode":"default","interview_turns":7,"review_iterations":2,"brief_quality":"complete","research_topics":3,"profile":"economy","phase_models":{"brief":"sonnet","research":"sonnet","plan":"sonnet","execute":"sonnet","review":"sonnet","continue":"sonnet"},"profile_source":"flag"}
{"ts":"2026-05-09T07:30:00.000Z","slug":"add-auth","mode":"default","scope":"local","dimensions":4,"agents_local":3,"agents_external":0,"profile":"economy","phase_models":{"brief":"sonnet","research":"sonnet","plan":"sonnet","execute":"sonnet","review":"sonnet","continue":"sonnet"},"parallel_agents":3,"external_research_enabled":false,"profile_source":"env"}
{"ts":"2026-05-09T08:00:00.000Z","slug":"add-auth","mode":"default","codebase_files":156,"agents_deployed":6,"deep_dives":2,"critic_verdict":"PASS","guardian_verdict":"ALIGNED","outcome":"execute","profile":"balanced","phase_models":{"brief":"sonnet","research":"sonnet","plan":"opus","execute":"sonnet","review":"opus","continue":"sonnet"},"parallel_agents":6,"profile_source":"default"}
{"ts":"2026-05-09T08:30:00.000Z","plan":"trekplan-add-auth.md","plan_type":"plan","mode":"execute","result":"completed","steps_total":12,"steps_passed":12,"steps_failed":0,"steps_skipped":0,"profile":"balanced","phase_models":{"brief":"sonnet","research":"sonnet","plan":"opus","execute":"sonnet","review":"opus","continue":"sonnet"},"profile_source":"inheritance"}
{"ts":"2026-05-09T09:00:00.000Z","slug":"add-auth","verdict":"ALLOW","reviewed_files_count":18,"mode":"default","duration_ms":4521,"profile":"premium","phase_models":{"brief":"opus","research":"opus","plan":"opus","execute":"opus","review":"opus","continue":"opus"},"profile_source":"flag"}

8
tests/fixtures/token-usage/sample-transcript.jsonl vendored
View file

@ -1,8 +0,0 @@
{"type":"user","message":{"role":"user","content":"hi"}}
{"type":"assistant","isSidechain":false,"requestId":"req_A","message":{"model":"claude-opus-4-8","usage":{"input_tokens":1,"output_tokens":1,"cache_creation_input_tokens":0,"cache_read_input_tokens":0}}}
{"type":"assistant","isSidechain":false,"requestId":"req_A","message":{"model":"claude-opus-4-8","usage":{"input_tokens":1000,"output_tokens":200,"cache_creation_input_tokens":400,"cache_read_input_tokens":5000}}}
{"type":"assistant","isSidechain":false,"requestId":"req_B","message":{"model":"claude-opus-4-8","usage":{"input_tokens":2000,"output_tokens":300,"cache_creation_input_tokens":600,"cache_read_input_tokens":1000}}}
{"type":"assistant","isSidechain":true,"requestId":"req_SIDE","message":{"model":"claude-opus-4-8","usage":{"input_tokens":9999,"output_tokens":9999,"cache_creation_input_tokens":9999,"cache_read_input_tokens":9999}}}
{"type":"system","content":"some system note with no usage"}
this is not valid json and must be skipped
{"type":"assistant","isSidechain":false,"requestId":"req_B","message":{"model":"claude-opus-4-8","usage":{"input_tokens":2000,"output_tokens":300,"cache_creation_input_tokens":600,"cache_read_input_tokens":1000}}}

2
tests/hooks/agent-cap.test.mjs
View file

@ -527,7 +527,7 @@ test('write snippet then real hook: the loop’s own writer arms the enforcement
});
// -----------------------------------------------------------------------
// Wiring — pattern from tests/hooks/hooks-json-stop-wired.test.mjs
// Wiring
// -----------------------------------------------------------------------
function invocationOf(h) {
return [h.command || '', ...(h.args || [])].join(' ').trim();

73
tests/hooks/hooks-json-stop-wired.test.mjs
View file

@ -1,73 +0,0 @@
// SC-13: hooks.json wires Stop event to otel-export.mjs
// HIGH-risk-mitigering — verify deterministic config-pinning (mønster fra
// tests/lib/doc-consistency.test.mjs).
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { readFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { dirname, resolve } from 'node:path';
const __dirname = dirname(fileURLToPath(import.meta.url));
const HOOKS_JSON_PATH = resolve(__dirname, '../../hooks/hooks.json');
function loadHooksJson() {
const raw = readFileSync(HOOKS_JSON_PATH, 'utf8');
return JSON.parse(raw);
}
// Hooks may use shell form ({command: "node X.mjs"}) or exec form
// ({command: "node", args: ["X.mjs"]}, CC 2.1.139). Normalize both to a single
// invocation string so assertions are form-agnostic.
function invocationOf(h) {
return [h.command || '', ...(h.args || [])].join(' ').trim();
}
test('hooks.json — Stop key exists with at least one entry', () => {
const cfg = loadHooksJson();
assert.ok(cfg.hooks, 'hooks.json mangler top-level "hooks" object');
assert.ok(Array.isArray(cfg.hooks.Stop), 'hooks.json mangler "Stop" array');
assert.ok(cfg.hooks.Stop.length >= 1, 'Stop array er tom — forventet ≥1 entry');
});
test('hooks.json — Stop entry refererer otel-export.mjs', () => {
const cfg = loadHooksJson();
const stopEntries = cfg.hooks.Stop;
const allInvocations = stopEntries.flatMap((entry) =>
(entry.hooks || []).map(invocationOf),
);
const hasOtelExport = allInvocations.some((cmd) => cmd.includes('otel-export.mjs'));
assert.ok(
hasOtelExport,
`ingen Stop-hook refererer otel-export.mjs. Funnet: ${JSON.stringify(allInvocations)}`,
);
});
test('hooks.json — Stop entry bruker ${CLAUDE_PLUGIN_ROOT}-substitusjon', () => {
const cfg = loadHooksJson();
const stopEntries = cfg.hooks.Stop;
const otelEntry = stopEntries
.flatMap((entry) => entry.hooks || [])
.find((h) => invocationOf(h).includes('otel-export.mjs'));
assert.ok(otelEntry, 'fant ikke otel-export-entry i Stop');
const otelInvocation = invocationOf(otelEntry);
assert.match(
otelInvocation,
/\$\{CLAUDE_PLUGIN_ROOT\}/,
'otel-export-invocation bruker ikke ${CLAUDE_PLUGIN_ROOT}-prefix — relative paths feiler i headless',
);
assert.match(
otelInvocation,
/^node\s+/,
'otel-export-invocation starter ikke med "node " — invocation-form ikke korrekt',
);
});
test('hooks.json — Stop entry har "type": "command"', () => {
const cfg = loadHooksJson();
const stopEntries = cfg.hooks.Stop;
const otelHook = stopEntries
.flatMap((entry) => entry.hooks || [])
.find((h) => invocationOf(h).includes('otel-export.mjs'));
assert.equal(otelHook.type, 'command', 'otel-export-hook mangler "type": "command"');
});

110
tests/hooks/otel-export-otlp.test.mjs
View file

@ -1,110 +0,0 @@
// tests/hooks/otel-export-otlp.test.mjs
// SC #13: lib/exporters/otlp-format.mjs returns OTLP/JSON v1.0 metrics payload
// with INTEGER (not string) enum constants and timeUnixNano as decimal STRING
// (JS precision-loss mitigation per research/01 + risk-assessor CRITICAL 2).
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { readFileSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import {
transformToOtlpJson,
AGG_TEMPORALITY_CUMULATIVE,
AGG_TEMPORALITY_DELTA,
AGG_TEMPORALITY_UNSPECIFIED,
} from '../../lib/exporters/otlp-format.mjs';
const __dirname = dirname(fileURLToPath(import.meta.url));
const FIXTURES = join(__dirname, '..', 'fixtures');
function loadJsonl(name) {
const text = readFileSync(join(FIXTURES, name), 'utf-8');
return text.trim().split('\n').filter(Boolean).map(l => JSON.parse(l));
}
test('SC #13: aggregationTemporality is INTEGER (typeof === "number"), not string', () => {
const records = loadJsonl('stats-sample.jsonl');
const payload = transformToOtlpJson(records);
// Find a sum metric (steps_passed is a counter)
const metrics = payload.resourceMetrics[0].scopeMetrics[0].metrics;
const sumMetric = metrics.find(m => 'sum' in m);
assert.ok(sumMetric, `expected at least one sum-metric in payload, got ${metrics.length} metrics`);
// CRITICAL: this assertion is the heart of SC #13 — typeof MUST be 'number'
assert.equal(typeof sumMetric.sum.aggregationTemporality, 'number',
`aggregationTemporality must be INTEGER (typeof number), got ${typeof sumMetric.sum.aggregationTemporality}`);
assert.equal(sumMetric.sum.aggregationTemporality, AGG_TEMPORALITY_CUMULATIVE);
assert.equal(sumMetric.sum.aggregationTemporality, 2);
});
test('SC #13: enum constants exported as integer literals (drift-pin)', () => {
assert.equal(typeof AGG_TEMPORALITY_UNSPECIFIED, 'number');
assert.equal(AGG_TEMPORALITY_UNSPECIFIED, 0);
assert.equal(typeof AGG_TEMPORALITY_DELTA, 'number');
assert.equal(AGG_TEMPORALITY_DELTA, 1);
assert.equal(typeof AGG_TEMPORALITY_CUMULATIVE, 'number');
assert.equal(AGG_TEMPORALITY_CUMULATIVE, 2);
});
test('SC #13: timeUnixNano is decimal STRING (typeof === "string"), JS precision-loss mitigation', () => {
const records = loadJsonl('stats-sample.jsonl');
const payload = transformToOtlpJson(records);
const metrics = payload.resourceMetrics[0].scopeMetrics[0].metrics;
// Pick first metric with a data point
const m = metrics.find(x => (x.sum?.dataPoints?.length || x.gauge?.dataPoints?.length) > 0);
const dp = (m.sum || m.gauge).dataPoints[0];
assert.equal(typeof dp.timeUnixNano, 'string',
`timeUnixNano must be decimal STRING, got ${typeof dp.timeUnixNano}: ${dp.timeUnixNano}`);
assert.equal(typeof dp.startTimeUnixNano, 'string');
// Should be a valid decimal-digit string
assert.match(dp.timeUnixNano, /^\d+$/);
});
test('SC #13: structural shape — resourceMetrics[].scopeMetrics[].metrics[] hierarchy', () => {
const records = loadJsonl('stats-sample.jsonl');
const payload = transformToOtlpJson(records);
assert.ok(Array.isArray(payload.resourceMetrics));
assert.ok(payload.resourceMetrics.length >= 1);
assert.ok(payload.resourceMetrics[0].resource);
assert.ok(Array.isArray(payload.resourceMetrics[0].scopeMetrics));
assert.ok(payload.resourceMetrics[0].scopeMetrics[0].scope);
assert.equal(payload.resourceMetrics[0].scopeMetrics[0].scope.name, 'voyage');
assert.ok(Array.isArray(payload.resourceMetrics[0].scopeMetrics[0].metrics));
});
test('Empty input: returns valid OTLP envelope with empty metrics array', () => {
const payload = transformToOtlpJson([]);
assert.ok(Array.isArray(payload.resourceMetrics));
assert.equal(payload.resourceMetrics[0].scopeMetrics[0].metrics.length, 0);
});
test('isSum heuristic: counter-named metrics get sum + isMonotonic; others get gauge', () => {
const records = [
{ _schema_id: 'test', ts: '2026-05-09T08:00:00.000Z', steps_total: 10 }, // counter
{ _schema_id: 'test', ts: '2026-05-09T08:00:00.000Z', cpu_pct: 42.5 }, // gauge
];
const payload = transformToOtlpJson(records);
const metrics = payload.resourceMetrics[0].scopeMetrics[0].metrics;
const totalMetric = metrics.find(m => m.name.endsWith('steps_total'));
const cpuMetric = metrics.find(m => m.name.endsWith('cpu_pct'));
assert.ok(totalMetric.sum, 'counter should have sum');
assert.equal(totalMetric.sum.isMonotonic, true);
assert.equal(typeof totalMetric.sum.aggregationTemporality, 'number');
assert.ok(cpuMetric.gauge, 'non-counter should have gauge');
assert.ok(!cpuMetric.sum, 'gauge should not have sum');
});
test('Allowlist redacted: callers strip command_excerpt before passing — verify nothing leaks', () => {
const record = {
_schema_id: 'post_bash_stats',
ts: '2026-05-09T08:00:00.000Z',
duration_ms: 152,
success: true,
};
const payload = transformToOtlpJson([record]);
const json = JSON.stringify(payload);
assert.ok(!json.includes('command_excerpt'));
assert.ok(!json.includes('session_id'));
// Should contain duration_ms metric
assert.match(json, /post_bash_stats\.duration_ms/);
});

135
tests/hooks/otel-export-textfile.test.mjs
View file

@ -1,135 +0,0 @@
// tests/hooks/otel-export-textfile.test.mjs
// SC #12: lib/exporters/textfile-format.mjs produces deterministic Prometheus
// text-format output that matches expected.prom byte-for-byte.
//
// To regenerate snapshot:
// node scripts/gen-expected-prom.mjs > tests/fixtures/expected.prom
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { readFileSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { transformToPrometheus, normalizeMetricName } from '../../lib/exporters/textfile-format.mjs';
import { transformToOtlpJson } from '../../lib/exporters/otlp-format.mjs';
import { applyFieldAllowlist } from '../../lib/exporters/field-allowlist.mjs';
const __dirname = dirname(fileURLToPath(import.meta.url));
const FIXTURES = join(__dirname, '..', 'fixtures');
function loadJsonl(name) {
const text = readFileSync(join(FIXTURES, name), 'utf-8');
return text.trim().split('\n').filter(Boolean).map(l => JSON.parse(l));
}
test('SC #12: stats-sample.jsonl → expected.prom snapshot byte-for-byte match', () => {
const records = loadJsonl('stats-sample.jsonl');
const actual = transformToPrometheus(records);
const expected = readFileSync(join(FIXTURES, 'expected.prom'), 'utf-8');
assert.equal(actual, expected,
`Snapshot drift detected. Regenerate via:\n` +
` node scripts/gen-expected-prom.mjs > tests/fixtures/expected.prom`);
});
test('Step 9: the four numeric STORM fields are metric families and effort is a label', () => {
const expected = readFileSync(join(FIXTURES, 'expected.prom'), 'utf-8');
for (const field of ['unique_sources', 'dimensions_baseline', 'conv_turns', 'empty_turns']) {
assert.match(
expected,
new RegExp(`^# TYPE voyage_trekresearch_${field} `, 'm'),
`${field} must appear as its own metric family — a numeric that never becomes a metric cannot be measured`,
);
}
// effort is a low-cardinality string: it must ride along as a LABEL, never
// as a metric family (a label is what makes high-vs-standard groupable).
assert.match(expected, /effort="[a-z]+"/, 'effort must be emitted as a label');
assert.doesNotMatch(
expected,
/^# TYPE voyage_trekresearch_effort /m,
'effort must not become a metric family',
);
});
test('empty-input handling: [] returns empty string (no headers)', () => {
assert.equal(transformToPrometheus([]), '');
assert.equal(transformToPrometheus(null), '');
assert.equal(transformToPrometheus(undefined), '');
});
test('allowlist-redaction: caller-redacted records (without command_excerpt/session_id) emit cleanly', () => {
// Simulate an allowlist-applied post-bash-stats record (command_excerpt + session_id removed)
const record = {
_schema_id: 'post_bash_stats',
ts: '2026-05-09T08:00:00.000Z',
duration_ms: 152,
success: true,
};
const out = transformToPrometheus([record]);
// Must NOT contain command_excerpt nor session_id (caller's responsibility, but verify)
assert.ok(!out.includes('command_excerpt'), 'command_excerpt leaked into output');
assert.ok(!out.includes('session_id'), 'session_id leaked into output');
// Must contain duration_ms metric
assert.match(out, /voyage_post_bash_stats_duration_ms/);
assert.match(out, / 152$/m);
// Boolean coerced to 1
assert.match(out, /voyage_post_bash_stats_success.* 1$/m);
});
test('NO client-side timestamps in output (per research/01 node_exporter#1284 mitigation)', () => {
const records = loadJsonl('stats-sample.jsonl');
const out = transformToPrometheus(records);
const lines = out.split('\n');
for (const line of lines) {
if (line.startsWith('#') || line === '') continue;
// Sample line format: metric{labels} value [timestamp]
// We must NOT have a trailing numeric timestamp after the value.
const parts = line.trim().split(' ');
assert.ok(parts.length === 2,
`Line has unexpected token count (timestamp leaked?): ${line}`);
}
});
test('normalizeMetricName: dots/dashes/spaces → underscore, lowercase, voyage_ prefix', () => {
assert.equal(normalizeMetricName('voyage.hook.duration_ms'), 'voyage_voyage_hook_duration_ms');
assert.equal(normalizeMetricName('Plan-Critic Verdict'), 'voyage_plan_critic_verdict');
assert.equal(normalizeMetricName('METRIC NAME'), 'voyage_metric_name');
});
test('SC#6: allowlisted token-usage record → Prometheus + OTLP metrics; PII absent (end-to-end)', () => {
// Raw record as written to token-usage-stats.jsonl (carries PII for upsert keying).
const raw = {
session_id: 'x',
transcript_path: '/p',
cwd: '/c',
ts: '2026-06-26T12:00:00.000Z',
scope: 'main-context',
model: 'claude-opus-4-8',
tokens_input: 12345,
tokens_output: 6789,
cost_usd: 0.42,
is_estimate: false,
price_table_version: '2026-06-26',
};
const rec = applyFieldAllowlist(raw, 'token-usage');
// Prometheus: numeric fields auto-promote to voyage_token_usage_* metrics.
const prom = transformToPrometheus([rec]);
assert.match(prom, /voyage_token_usage_tokens_input\b/);
assert.match(prom, /voyage_token_usage_cost_usd\b/);
// CWE-212: PII must never appear in exporter output.
assert.ok(!prom.includes('session_id'), 'session_id leaked into Prometheus output');
assert.ok(!prom.includes('transcript_path'), 'transcript_path leaked into Prometheus output');
assert.ok(!prom.includes('/p'), 'transcript_path value leaked into Prometheus output');
// OTLP parity: dot-separated, dash-preserved naming (NOT the Prometheus underscore form).
const otlp = JSON.stringify(transformToOtlpJson([rec]));
assert.ok(otlp.includes('voyage.token-usage.tokens_input'), 'OTLP token metric missing');
assert.ok(!otlp.includes('transcript_path'), 'transcript_path leaked into OTLP output');
});
test('determinism: identical input produces identical output (sorted keys)', () => {
const records = loadJsonl('stats-sample.jsonl');
const out1 = transformToPrometheus(records);
const out2 = transformToPrometheus([...records]);
assert.equal(out1, out2);
});

147
tests/hooks/otel-export-token-capture.test.mjs
View file

@ -1,147 +0,0 @@
// tests/hooks/otel-export-token-capture.test.mjs
// SKAL-2 Step 4 — opt-in main-context token/cost capture folded into the Stop
// hook (otel-export.mjs), gated by VOYAGE_TOKEN_METER.
//
// Fail-open contract: any error → exit 0, no token file, Stop never blocked.
// Pattern: tests/hooks/otel-export.test.mjs (setupDataDir + runHookWithEnv).
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { mkdtempSync, rmSync, existsSync, readFileSync, writeFileSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { tmpdir } from 'node:os';
import { fileURLToPath } from 'node:url';
import { runHookWithEnv } from '../helpers/hook-helper.mjs';
const HERE = dirname(fileURLToPath(import.meta.url));
const HOOK_PATH = join(HERE, '..', '..', 'hooks', 'scripts', 'otel-export.mjs');
const FIXTURE = join(HERE, '..', 'fixtures', 'token-usage', 'sample-transcript.jsonl');
const TOKEN_FILE = 'token-usage-stats.jsonl';
const mkDataDir = () => mkdtempSync(join(tmpdir(), 'voyage-token-capture-'));
const stdinFor = (extra = {}) => JSON.stringify({ transcript_path: FIXTURE, session_id: 'sess-1', ...extra });
const readToken = (dir) => readFileSync(join(dir, TOKEN_FILE), 'utf-8').trim().split('\n').filter(Boolean).map(JSON.parse);
test('VOYAGE_TOKEN_METER=1 → writes token-usage-stats.jsonl with expected totals + scope:main-context', async () => {
const dir = mkDataDir();
try {
const r = await runHookWithEnv(HOOK_PATH, stdinFor(), {
VOYAGE_TOKEN_METER: '1',
VOYAGE_EXPORT_MODE: 'off',
CLAUDE_PLUGIN_DATA: dir,
});
assert.equal(r.code, 0);
assert.equal(existsSync(join(dir, TOKEN_FILE)), true, 'token file must be written');
const recs = readToken(dir);
assert.equal(recs.length, 1);
const rec = recs[0];
// Fixture totals: dedup-by-requestId + sidechain-exclusion (verified in Step 1).
assert.equal(rec.tokens_input, 3000);
assert.equal(rec.tokens_output, 500);
assert.equal(rec.tokens_cache_creation, 1000);
assert.equal(rec.tokens_cache_read, 6000);
assert.equal(rec.scope, 'main-context');
assert.equal(rec.model, 'claude-opus-4-8');
assert.ok(Math.abs(rec.cost_usd - 0.03675) < 1e-9, `cost ${rec.cost_usd}`);
assert.equal(rec.session_id, 'sess-1');
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test('UPSERT — running twice for the same session_id keeps ONE line (not two)', async () => {
const dir = mkDataDir();
try {
const env = { VOYAGE_TOKEN_METER: '1', VOYAGE_EXPORT_MODE: 'off', CLAUDE_PLUGIN_DATA: dir };
await runHookWithEnv(HOOK_PATH, stdinFor(), env);
const r2 = await runHookWithEnv(HOOK_PATH, stdinFor(), env);
assert.equal(r2.code, 0);
const recs = readToken(dir);
assert.equal(recs.length, 1, 'upsert must NOT append a second line for the same session');
assert.equal(recs[0].tokens_input, 3000);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test('UPSERT — distinct session_ids append (one line per session)', async () => {
const dir = mkDataDir();
try {
const env = { VOYAGE_TOKEN_METER: '1', VOYAGE_EXPORT_MODE: 'off', CLAUDE_PLUGIN_DATA: dir };
await runHookWithEnv(HOOK_PATH, stdinFor({ session_id: 'A' }), env);
await runHookWithEnv(HOOK_PATH, stdinFor({ session_id: 'B' }), env);
const recs = readToken(dir);
assert.equal(recs.length, 2);
assert.deepEqual(recs.map(r => r.session_id).sort(), ['A', 'B']);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test('OPT-IN — VOYAGE_TOKEN_METER unset → NO token file written', async () => {
const dir = mkDataDir();
try {
const r = await runHookWithEnv(HOOK_PATH, stdinFor(), {
VOYAGE_TOKEN_METER: '', // explicit empty mimics unset
VOYAGE_EXPORT_MODE: 'off',
CLAUDE_PLUGIN_DATA: dir,
});
assert.equal(r.code, 0);
assert.equal(existsSync(join(dir, TOKEN_FILE)), false, 'no token file without opt-in');
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test('FAIL-OPEN — malformed stdin → exit 0, no token file, no throw', async () => {
const dir = mkDataDir();
try {
const r = await runHookWithEnv(HOOK_PATH, 'not valid json {{{', {
VOYAGE_TOKEN_METER: '1',
VOYAGE_EXPORT_MODE: 'off',
CLAUDE_PLUGIN_DATA: dir,
});
assert.equal(r.code, 0);
assert.equal(existsSync(join(dir, TOKEN_FILE)), false);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test('FAIL-OPEN — missing transcript file → exit 0, no token file', async () => {
const dir = mkDataDir();
try {
const r = await runHookWithEnv(
HOOK_PATH,
JSON.stringify({ transcript_path: '/no/such/transcript.jsonl', session_id: 'x' }),
{ VOYAGE_TOKEN_METER: '1', VOYAGE_EXPORT_MODE: 'off', CLAUDE_PLUGIN_DATA: dir },
);
assert.equal(r.code, 0);
assert.equal(existsSync(join(dir, TOKEN_FILE)), false);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test('capture coexists with export — VOYAGE_TOKEN_METER=1 + textfile mode writes BOTH', async () => {
const dir = mkDataDir();
try {
// Seed a stats file so the export path has something to write.
writeFileSync(join(dir, 'trekplan-stats.jsonl'),
JSON.stringify({ ts: '2026-06-26T08:00:00.000Z', slug: 't', mode: 'default', codebase_files: 10, profile: 'premium' }) + '\n');
const r = await runHookWithEnv(HOOK_PATH, stdinFor(), {
VOYAGE_TOKEN_METER: '1',
VOYAGE_EXPORT_MODE: 'textfile',
CLAUDE_PLUGIN_DATA: dir,
});
assert.equal(r.code, 0);
assert.equal(existsSync(join(dir, TOKEN_FILE)), true, 'token file written');
assert.equal(existsSync(join(dir, 'voyage.prom')), true, 'export still completed');
// The exported textfile should now also carry the token metric (STATS_FILES wiring).
const prom = readFileSync(join(dir, 'voyage.prom'), 'utf-8');
assert.match(prom, /voyage_token_usage_tokens_input/);
assert.ok(!prom.includes('sess-1'), 'session_id must not leak into export (CWE-212)');
} finally {
rmSync(dir, { recursive: true, force: true });
}
});

349
tests/hooks/otel-export-validators.test.mjs
View file

@ -1,349 +0,0 @@
// tests/hooks/otel-export-validators.test.mjs
// Step 11 validators: path, endpoint, field-allowlist.
// CWE-22, CWE-918, CWE-212 mitigation.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { validateTextfilePath, FORBIDDEN_PREFIXES } from '../../lib/exporters/path-validator.mjs';
import { validateOtlpEndpoint } from '../../lib/exporters/endpoint-validator.mjs';
import {
applyFieldAllowlist,
POST_BASH_STATS_ALLOWED,
EVENT_EMIT_PAYLOAD_ALLOWED,
TOKEN_USAGE_ALLOWED,
TREKRESEARCH_ALLOWED,
} from '../../lib/exporters/field-allowlist.mjs';
// ---- path-validator: CWE-22 mitigation -------------------------------------
test('path-validator: rejects ../etc/passwd traversal (CWE-22)', () => {
const r = validateTextfilePath('../../etc/passwd');
assert.equal(r.valid, false);
assert.ok(r.errors.find(e => e.code === 'PATH_TRAVERSAL'));
});
test('path-validator: rejects /etc/voyage.prom (forbidden system prefix)', () => {
const r = validateTextfilePath('/etc/voyage.prom');
assert.equal(r.valid, false);
// Either forbidden-system or parent-missing (both are deny-paths)
const denied = r.errors.find(e =>
e.code === 'PATH_FORBIDDEN_SYSTEM' || e.code === 'PATH_PARENT_MISSING');
assert.ok(denied, `expected deny, got: ${JSON.stringify(r.errors)}`);
});
test('path-validator: rejects ~ home shorthand', () => {
const r = validateTextfilePath('~/voyage.prom');
assert.equal(r.valid, false);
assert.ok(r.errors.find(e => e.code === 'PATH_HOME_SHORTHAND'));
});
test('path-validator: accepts path under allowedRoots', () => {
const tmp = mkdtempSync(join(tmpdir(), 'voyage-path-allow-'));
try {
const target = join(tmp, 'voyage.prom');
const r = validateTextfilePath(target, { allowedRoots: [tmp] });
assert.equal(r.valid, true, JSON.stringify(r.errors));
assert.match(r.parsed.path, /voyage\.prom$/);
} finally {
rmSync(tmp, { recursive: true, force: true });
}
});
test('path-validator: rejects path outside allowedRoots', () => {
const tmp = mkdtempSync(join(tmpdir(), 'voyage-path-deny-'));
const otherTmp = mkdtempSync(join(tmpdir(), 'voyage-path-other-'));
try {
const target = join(otherTmp, 'voyage.prom');
const r = validateTextfilePath(target, { allowedRoots: [tmp] });
assert.equal(r.valid, false);
assert.ok(r.errors.find(e => e.code === 'PATH_OUT_OF_ALLOWLIST'));
} finally {
rmSync(tmp, { recursive: true, force: true });
rmSync(otherTmp, { recursive: true, force: true });
}
});
test('path-validator: FORBIDDEN_PREFIXES exports drift-pin', () => {
// Ensure all the high-risk system paths are present
for (const prefix of ['/etc/', '/proc/', '/sys/', '/var/', '/usr/']) {
assert.ok(FORBIDDEN_PREFIXES.includes(prefix),
`FORBIDDEN_PREFIXES missing critical path: ${prefix}`);
}
});
// ---- endpoint-validator: CWE-918 mitigation -------------------------------
test('endpoint-validator: rejects http://169.254.169.254/ — PERMANENTLY blocked (CWE-918 cloud metadata)', () => {
const r = validateOtlpEndpoint('http://169.254.169.254/v1/metrics', { env: {} });
assert.equal(r.valid, false);
assert.ok(r.errors.find(e => e.code === 'ENDPOINT_HARD_BLOCKED'));
});
test('endpoint-validator: 169.254.169.254 stays blocked EVEN WITH VOYAGE_OTEL_ALLOW_PRIVATE=1', () => {
// Cloud metadata service is qualitatively different from RFC-1918 home-lab
// access — operator-trust is NOT extended here. AWS/GCP/Azure metadata
// exposes IAM credentials and can compromise the entire cloud account.
const r = validateOtlpEndpoint('http://169.254.169.254/v1/metrics',
{ env: { VOYAGE_OTEL_ALLOW_PRIVATE: '1' } });
assert.equal(r.valid, false, 'cloud metadata MUST stay blocked even with opt-in');
assert.ok(r.errors.find(e => e.code === 'ENDPOINT_HARD_BLOCKED'));
});
test('endpoint-validator: AliCloud metadata 100.100.100.200 PERMANENTLY blocked', () => {
const r = validateOtlpEndpoint('http://100.100.100.200/latest/meta-data',
{ env: { VOYAGE_OTEL_ALLOW_PRIVATE: '1' } });
assert.equal(r.valid, false);
assert.ok(r.errors.find(e => e.code === 'ENDPOINT_HARD_BLOCKED'));
});
test('endpoint-validator: metadata.google.internal hostname PERMANENTLY blocked', () => {
const r = validateOtlpEndpoint('http://metadata.google.internal/computeMetadata/v1',
{ env: { VOYAGE_OTEL_ALLOW_PRIVATE: '1' } });
assert.equal(r.valid, false);
assert.ok(r.errors.find(e => e.code === 'ENDPOINT_HARD_BLOCKED'));
});
test('endpoint-validator: rejects http://example.com/ (requires https)', () => {
const r = validateOtlpEndpoint('http://example.com/v1/metrics', { env: {} });
assert.equal(r.valid, false);
assert.ok(r.errors.find(e => e.code === 'ENDPOINT_HTTPS_REQUIRED'));
});
test('endpoint-validator: rejects http://localhost without VOYAGE_OTEL_ALLOW_PRIVATE', () => {
const r = validateOtlpEndpoint('http://localhost:4318/v1/metrics', { env: {} });
assert.equal(r.valid, false);
assert.ok(r.errors.find(e => e.code === 'ENDPOINT_LOOPBACK_REJECTED'));
});
test('endpoint-validator: accepts http://localhost when VOYAGE_OTEL_ALLOW_PRIVATE=1 (home-lab opt-in)', () => {
const r = validateOtlpEndpoint('http://localhost:4318/v1/metrics',
{ env: { VOYAGE_OTEL_ALLOW_PRIVATE: '1' } });
assert.equal(r.valid, true, JSON.stringify(r.errors));
assert.equal(r.parsed.isPrivate, true);
});
test('endpoint-validator: accepts https://example.com/v1/metrics (public)', () => {
const r = validateOtlpEndpoint('https://otel.example.com/v1/metrics', { env: {} });
assert.equal(r.valid, true, JSON.stringify(r.errors));
assert.equal(r.parsed.isPrivate, false);
});
test('endpoint-validator: rejects RFC-1918 192.168.1.1 without opt-in', () => {
const r = validateOtlpEndpoint('http://192.168.1.1:4318/v1/metrics', { env: {} });
assert.equal(r.valid, false);
assert.ok(r.errors.find(e => e.code === 'ENDPOINT_RFC1918_REJECTED'));
});
// IPv4-mapped IPv6 bypass (S21): `::ffff:a.b.c.d` literals render as
// `::ffff:HHHH:HHHH` and route to the embedded IPv4. The host must be
// canonicalized to its embedded IPv4 BEFORE classification, or loopback /
// RFC-1918 / link-local / cloud-metadata guards are all bypassable over https.
test('endpoint-validator: rejects IPv4-mapped IPv6 loopback [::ffff:127.0.0.1] (S21 bypass)', () => {
const r = validateOtlpEndpoint('https://[::ffff:127.0.0.1]/v1/metrics', { env: {} });
assert.equal(r.valid, false, 'IPv4-mapped loopback must be rejected');
assert.ok(r.errors.find(e => e.code === 'ENDPOINT_LOOPBACK_REJECTED'));
});
test('endpoint-validator: rejects IPv4-mapped IPv6 RFC-1918 [::ffff:192.168.0.5] (S21 bypass)', () => {
const r = validateOtlpEndpoint('https://[::ffff:192.168.0.5]/v1/metrics', { env: {} });
assert.equal(r.valid, false, 'IPv4-mapped RFC-1918 must be rejected');
assert.ok(r.errors.find(e => e.code === 'ENDPOINT_RFC1918_REJECTED'));
});
test('endpoint-validator: IPv4-mapped metadata [::ffff:169.254.169.254] HARD_BLOCKED even with opt-in (S21 bypass)', () => {
const r = validateOtlpEndpoint('https://[::ffff:169.254.169.254]/latest/meta-data',
{ env: { VOYAGE_OTEL_ALLOW_PRIVATE: '1' } });
assert.equal(r.valid, false, 'IPv4-mapped cloud-metadata MUST stay hard-blocked');
assert.ok(r.errors.find(e => e.code === 'ENDPOINT_HARD_BLOCKED'));
});
test('endpoint-validator: accepts IPv4-mapped IPv6 public [::ffff:8.8.8.8] over https (no over-block)', () => {
const r = validateOtlpEndpoint('https://[::ffff:8.8.8.8]/v1/metrics', { env: {} });
assert.equal(r.valid, true, JSON.stringify(r.errors));
assert.equal(r.parsed.isPrivate, false);
});
test('endpoint-validator: rejects empty / non-string', () => {
assert.equal(validateOtlpEndpoint('').valid, false);
assert.equal(validateOtlpEndpoint(null).valid, false);
assert.equal(validateOtlpEndpoint(undefined).valid, false);
});
// ---- field-allowlist: CWE-212 mitigation -----------------------------------
test('field-allowlist: post-bash-stats DROPS command_excerpt + session_id (CWE-212)', () => {
const record = {
ts: '2026-05-09T08:00:00.000Z',
session_id: 'uuid-12345',
command_excerpt: 'git clone https://example.com/secret/repo',
duration_ms: 152,
success: true,
};
const out = applyFieldAllowlist(record, 'post-bash-stats');
assert.equal('command_excerpt' in out, false, 'command_excerpt MUST be stripped');
assert.equal('session_id' in out, false, 'session_id MUST be stripped');
assert.equal(out.duration_ms, 152);
assert.equal(out.success, true);
assert.equal(out._schema_id, 'post-bash-stats');
});
test('field-allowlist: trekplan DROPS task / project_dir / brief_path (PII)', () => {
const record = {
ts: '2026-05-09T08:00:00.000Z',
task: 'private user prose with PII',
slug: 'add-auth',
project_dir: '/home/user/secret/project',
brief_path: '/home/user/secret/brief.md',
codebase_files: 156,
profile: 'premium',
};
const out = applyFieldAllowlist(record, 'trekplan');
assert.equal('task' in out, false);
assert.equal('project_dir' in out, false);
assert.equal('brief_path' in out, false);
assert.equal(out.slug, 'add-auth');
assert.equal(out.codebase_files, 156);
assert.equal(out.profile, 'premium');
});
// M5: 2045432 added intent_approved to the trekbrief allowlist and no test
// guarded it — removing the line left the suite green.
test('field-allowlist: trekbrief KEEPS intent_approved (veikart steg 1) and drops task / project_dir', () => {
const record = {
ts: '2026-09-23T10:00:00.000Z',
task: 'private user prose',
slug: 'dark-mode-toggle',
project_dir: '/home/user/secret/project',
intent_approved: true,
};
const out = applyFieldAllowlist(record, 'trekbrief');
assert.equal(out.intent_approved, true);
assert.equal(out.slug, 'dark-mode-toggle');
assert.equal('task' in out, false);
assert.equal('project_dir' in out, false);
assert.equal(applyFieldAllowlist({ ...record, intent_approved: false }, 'trekbrief').intent_approved, false);
});
test('field-allowlist: trekbrief DROPS the order fields (coordination metadata) and still KEEPS intent_approved', () => {
const out = applyFieldAllowlist({
ts: '2026-10-04T10:00:00.000Z', slug: 's', intent_approved: true,
order_id: 'o-1', order_claimed_at: '2026-10-04T09:00:00Z', order_reason: null,
}, 'trekbrief');
for (const k of ['order_id', 'order_claimed_at', 'order_reason']) assert.equal(k in out, false, `${k} must never be exported`);
assert.equal(out.intent_approved, true);
});
test('field-allowlist: event-emit applies sub-allowlist to payload', () => {
const record = {
ts: '2026-05-09T08:00:00.000Z',
event: 'main-merge-gate',
known_event: true,
payload: {
profile: 'balanced',
profile_source: 'inheritance',
command_excerpt: 'should be stripped from payload',
raw_user_prose: 'should be stripped',
},
};
const out = applyFieldAllowlist(record, 'event-emit');
assert.equal(out.event, 'main-merge-gate');
assert.equal(out.payload.profile, 'balanced');
assert.equal(out.payload.profile_source, 'inheritance');
assert.equal('command_excerpt' in out.payload, false);
assert.equal('raw_user_prose' in out.payload, false);
});
test('field-allowlist: unknown schema-type returns conservative {ts, _schema_id} only', () => {
const out = applyFieldAllowlist(
{ ts: '2026-05-09T08:00:00.000Z', sensitive: 'secret' },
'totally-unknown-schema',
);
assert.equal('sensitive' in out, false);
assert.equal(out.ts, '2026-05-09T08:00:00.000Z');
assert.equal(out._schema_id, 'totally-unknown-schema');
});
test('field-allowlist: Object.freeze on allowlists (drift-pin)', () => {
assert.equal(Object.isFrozen(POST_BASH_STATS_ALLOWED), true,
'POST_BASH_STATS_ALLOWED must be frozen — runtime mutation prevention');
assert.equal(Object.isFrozen(EVENT_EMIT_PAYLOAD_ALLOWED), true);
assert.equal(Object.isFrozen(TOKEN_USAGE_ALLOWED), true,
'TOKEN_USAGE_ALLOWED must be frozen — runtime mutation prevention');
});
// ---- token-usage allowlist (SKAL-2, CWE-212) -------------------------------
test('field-allowlist: token-usage INCLUDES numeric/label fields, EXCLUDES session_id/transcript_path/cwd (two-sided)', () => {
const record = {
ts: '2026-06-26T12:00:00.000Z',
session_id: 'uuid-secret',
transcript_path: '/Users/ktg/.claude/projects/x/sesn.jsonl',
cwd: '/Users/ktg/secret/project',
scope: 'main-context',
model: 'claude-opus-4-8',
tokens_input: 12345,
tokens_output: 6789,
tokens_cache_creation: 400,
tokens_cache_read: 5000,
cost_usd: 0.42,
is_estimate: false,
price_table_version: '2026-06-26',
};
const out = applyFieldAllowlist(record, 'token-usage');
// INCLUDED (numeric + low-cardinality labels)
for (const k of ['tokens_input', 'tokens_output', 'tokens_cache_creation',
'tokens_cache_read', 'cost_usd', 'is_estimate', 'price_table_version', 'scope', 'model']) {
assert.equal(k in out, true, `${k} MUST be allowlisted`);
}
assert.equal(out.tokens_input, 12345);
assert.equal(out._schema_id, 'token-usage');
// EXCLUDED (CWE-212 boundary)
assert.equal('session_id' in out, false, 'session_id MUST be stripped (CWE-212)');
assert.equal('transcript_path' in out, false, 'transcript_path MUST be stripped (CWE-212)');
assert.equal('cwd' in out, false, 'cwd MUST be stripped (CWE-212)');
});
// ---- trekresearch allowlist: the `engine` field ----------------------------
test('field-allowlist: trekresearch INCLUDES engine, EXCLUDES question/project_dir/brief_path (two-sided)', () => {
const record = {
ts: '2026-08-09T12:00:00.000Z',
question: 'which retrieval strategy survives contradiction?',
mode: 'default',
scope: 'both',
engine: 'deep-research',
slug: 'storm-upgrade',
project_dir: '/Users/ktg/secret/project',
brief_path: '/Users/ktg/secret/project/brief.md',
dimensions: 4,
agents_local: 7,
agents_external: 4,
gemini_used: false,
confidence: 0.82,
contradictions: 1,
open_questions: 3,
};
const out = applyFieldAllowlist(record, 'trekresearch');
// INCLUDED — low-cardinality label, emitted (trekresearch.md:533) and
// promised in prose (:570-572); it was silently dropped before this pin.
assert.equal('engine' in out, true, 'engine MUST be allowlisted — it is emitted and documented');
assert.equal(out.engine, 'deep-research');
assert.equal(out._schema_id, 'trekresearch');
// EXCLUDED (CWE-212 boundary)
assert.equal('question' in out, false, 'question MUST be stripped (prose, CWE-212)');
assert.equal('project_dir' in out, false, 'project_dir MUST be stripped (path, CWE-212)');
assert.equal('brief_path' in out, false, 'brief_path MUST be stripped (path, CWE-212)');
});
test('field-allowlist: TREKRESEARCH_ALLOWED is frozen (drift-pin)', () => {
assert.equal(Object.isFrozen(TREKRESEARCH_ALLOWED), true,
'TREKRESEARCH_ALLOWED must be frozen — runtime mutation prevention');
});
test('field-allowlist: null/undefined record handled safely', () => {
assert.deepEqual(applyFieldAllowlist(null, 'trekplan'), {});
assert.deepEqual(applyFieldAllowlist(undefined, 'trekplan'), {});
});

134
tests/hooks/otel-export.test.mjs
View file

@ -1,134 +0,0 @@
// tests/hooks/otel-export.test.mjs
// SC #14: Stop-hook orchestration — opt-in via VOYAGE_EXPORT_MODE.
// Fail-soft contract: any error → exit 0, [voyage] stderr, no Stop blocking.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { mkdtempSync, rmSync, existsSync, readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { tmpdir } from 'node:os';
import { fileURLToPath } from 'node:url';
import { runHookWithEnv } from '../helpers/hook-helper.mjs';
import { fastestWithin } from '../helpers/timing.mjs';
const __dirname = dirname(fileURLToPath(import.meta.url));
const HOOK_PATH = join(__dirname, '..', '..', 'hooks', 'scripts', 'otel-export.mjs');
function setupDataDir() {
const dir = mkdtempSync(join(tmpdir(), 'voyage-otel-data-'));
// Seed minimal stats files
writeFileSync(join(dir, 'trekplan-stats.jsonl'),
JSON.stringify({ ts: '2026-05-09T08:00:00.000Z', slug: 'test', mode: 'default', codebase_files: 100, profile: 'premium' }) + '\n');
return dir;
}
test('SC #14: VOYAGE_EXPORT_MODE=off → silent exit 0, no file written', async () => {
const dataDir = setupDataDir();
try {
const target = join(dataDir, 'voyage.prom');
const r = await runHookWithEnv(HOOK_PATH, '{}', {
VOYAGE_EXPORT_MODE: 'off',
CLAUDE_PLUGIN_DATA: dataDir,
});
assert.equal(r.code, 0);
assert.equal(existsSync(target), false, 'voyage.prom should NOT be written in off-mode');
assert.equal(r.stderr, '', 'no stderr expected');
} finally {
rmSync(dataDir, { recursive: true, force: true });
}
});
test('SC #14: VOYAGE_EXPORT_MODE unset → silent exit 0 (default off behavior)', async () => {
const dataDir = setupDataDir();
try {
const target = join(dataDir, 'voyage.prom');
const r = await runHookWithEnv(HOOK_PATH, '{}', {
CLAUDE_PLUGIN_DATA: dataDir,
VOYAGE_EXPORT_MODE: '', // explicit empty (mimics unset)
});
assert.equal(r.code, 0);
assert.equal(existsSync(target), false);
} finally {
rmSync(dataDir, { recursive: true, force: true });
}
});
test('SC #14: VOYAGE_EXPORT_MODE=textfile + valid CLAUDE_PLUGIN_DATA → writes voyage.prom', async () => {
const dataDir = setupDataDir();
try {
const target = join(dataDir, 'voyage.prom');
const r = await runHookWithEnv(HOOK_PATH, '{}', {
VOYAGE_EXPORT_MODE: 'textfile',
CLAUDE_PLUGIN_DATA: dataDir,
});
assert.equal(r.code, 0);
assert.equal(existsSync(target), true, `voyage.prom should be written; stderr: ${r.stderr}`);
const text = readFileSync(target, 'utf-8');
assert.match(text, /# HELP /);
assert.match(text, /# TYPE /);
assert.match(text, /voyage_trekplan_codebase_files/);
} finally {
rmSync(dataDir, { recursive: true, force: true });
}
});
test('SC #14: VOYAGE_EXPORT_MODE=invalid → stderr [voyage] warning + exit 0 (NOT blocking)', async () => {
const dataDir = setupDataDir();
try {
const r = await runHookWithEnv(HOOK_PATH, '{}', {
VOYAGE_EXPORT_MODE: 'banana',
CLAUDE_PLUGIN_DATA: dataDir,
});
assert.equal(r.code, 0, 'invalid mode MUST NOT block Stop');
assert.match(r.stderr, /\[voyage\]/);
assert.match(r.stderr, /banana/);
} finally {
rmSync(dataDir, { recursive: true, force: true });
}
});
test('SC #14: VOYAGE_EXPORT_MODE=otlp + invalid endpoint → stderr [voyage] warn + exit 0', async () => {
const dataDir = setupDataDir();
try {
const r = await runHookWithEnv(HOOK_PATH, '{}', {
VOYAGE_EXPORT_MODE: 'otlp',
VOYAGE_OTEL_ENDPOINT: 'http://example.com/v1/metrics', // public-http rejected
CLAUDE_PLUGIN_DATA: dataDir,
});
assert.equal(r.code, 0);
assert.match(r.stderr, /\[voyage\]/);
} finally {
rmSync(dataDir, { recursive: true, force: true });
}
});
test('SC #14: tail-latency for textfile mode < 200ms (NFR)', async () => {
const dataDir = setupDataDir();
try {
// The fastest of up to three cold runs against the same 800 ms limit
// (tests/helpers/timing.mjs): load only adds time, so a real slowness
// still fails every run, and one run slowed by a busy machine does not.
const t = await fastestWithin(800, async () => {
const start = performance.now();
const r = await runHookWithEnv(HOOK_PATH, '{}', {
VOYAGE_EXPORT_MODE: 'textfile',
CLAUDE_PLUGIN_DATA: dataDir,
});
assert.equal(r.code, 0);
return performance.now() - start;
});
// 200ms NFR with extra headroom for cold-start node spawn (~100ms typical)
assert.ok(t.ok,
`textfile export tail-latency too slow: fastest ${t.fastest.toFixed(0)}ms of ${t.runs.map((x) => x.toFixed(0)).join('/')}ms (NFR <200ms in-process; <800ms allowed for cold spawn)`);
} finally {
rmSync(dataDir, { recursive: true, force: true });
}
});
test('SC #14: missing CLAUDE_PLUGIN_DATA → silent exit 0', async () => {
const r = await runHookWithEnv(HOOK_PATH, '{}', {
VOYAGE_EXPORT_MODE: 'textfile',
CLAUDE_PLUGIN_DATA: '',
});
assert.equal(r.code, 0);
});

59
tests/integration/observability-compose.test.mjs
View file

@ -1,59 +0,0 @@
// SC #16 — skip-if-no-docker compose-config validation.
// First test in tests/integration/ — establishes the skip-on-missing-tool
// pattern voyage uses for environment-dependent integration tests.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { execFileSync, spawnSync } from 'node:child_process';
import { fileURLToPath } from 'node:url';
import { dirname, resolve } from 'node:path';
const __dirname = dirname(fileURLToPath(import.meta.url));
const REPO_ROOT = resolve(__dirname, '../..');
const COMPOSE_FILE = resolve(REPO_ROOT, 'examples/observability/docker-compose.yml');
const dockerAvailable = (() => {
try {
execFileSync('docker', ['info'], { stdio: 'ignore' });
return true;
} catch {
return false;
}
})();
test(
'compose config parses and contains expected services',
{ skip: !dockerAvailable && 'Docker not installed' },
() => {
const r = spawnSync(
'docker',
['compose', '-f', COMPOSE_FILE, 'config'],
{ encoding: 'utf8' },
);
assert.equal(r.status, 0, `docker compose config exited ${r.status}: ${r.stderr}`);
assert.match(r.stdout, /otel-collector/, 'otel-collector service missing');
assert.match(r.stdout, /prometheus/, 'prometheus service missing');
assert.match(r.stdout, /grafana/, 'grafana service missing');
assert.match(r.stdout, /node-exporter/, 'node-exporter service missing');
},
);
test(
'compose config pins required image versions',
{ skip: !dockerAvailable && 'Docker not installed' },
() => {
const r = spawnSync(
'docker',
['compose', '-f', COMPOSE_FILE, 'config'],
{ encoding: 'utf8' },
);
assert.equal(r.status, 0);
assert.match(r.stdout, /prom\/prometheus:v3\.0\.1/, 'prometheus pin missing');
assert.match(r.stdout, /grafana\/grafana:11\.4\.0/, 'grafana pin missing');
assert.match(
r.stdout,
/otel\/opentelemetry-collector-contrib:0\.115\.0/,
'otel-collector pin missing',
);
},
);

84
tests/lib/cache-analyzer.test.mjs
View file

@ -1,84 +0,0 @@
// tests/lib/cache-analyzer.test.mjs
// SKAL-2 Step 3 — token/cost aggregation in cache-analyzer's summarize().
// Also the first test coverage for cache-analyzer (previously zero-coverage):
// includes a regression guard that the pre-existing duration/event-name fields
// still compute correctly on mixed input.
//
// Hermetic: no LLM, network, time, or randomness. Idiom:
// tests/lib/token-usage.test.mjs (pure-module node:test style).
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { summarize } from '../../lib/stats/cache-analyzer.mjs';
const jsonl = (objs) => objs.map(o => JSON.stringify(o));
// ---- token/cost aggregation ------------------------------------------------
test('summarize — two token-bearing records → summed totals + cost', () => {
const s = summarize(jsonl([
{ ts: '2026-06-26T12:00:00Z', session_id: 'S1', scope: 'main-context', tokens_input: 1000, tokens_output: 200, tokens_cache_creation: 400, tokens_cache_read: 5000, cost_usd: 0.03675 },
{ ts: '2026-06-26T13:00:00Z', session_id: 'S2', scope: 'main-context', tokens_input: 2000, tokens_output: 300, tokens_cache_creation: 600, tokens_cache_read: 1000, cost_usd: 0.05 },
]));
assert.equal(s.total_tokens_input, 3000);
assert.equal(s.total_tokens_output, 500);
assert.equal(s.total_tokens_cache_creation, 1000);
assert.equal(s.total_tokens_cache_read, 6000);
assert.equal(s.sessions_with_tokens, 2);
assert.ok(Math.abs(s.total_cost_usd - 0.08675) < 1e-9, `cost ${s.total_cost_usd}`);
});
test('summarize — cost_usd:null counts the session but is excluded from total_cost_usd', () => {
const s = summarize(jsonl([
{ session_id: 'S1', tokens_input: 100, cost_usd: 0.01 },
{ session_id: 'S2', tokens_input: 200, cost_usd: null, is_estimate: true },
]));
assert.equal(s.sessions_with_tokens, 2, 'null-cost session must still be counted');
assert.equal(s.total_tokens_input, 300);
assert.ok(Math.abs(s.total_cost_usd - 0.01) < 1e-9, `cost ${s.total_cost_usd}`);
});
test('summarize — no token records → token totals stay zero (additive default)', () => {
const s = summarize(jsonl([
{ event: 'main-merge-gate', duration_ms: 100 },
]));
assert.equal(s.total_tokens_input, 0);
assert.equal(s.total_cost_usd, 0);
assert.equal(s.sessions_with_tokens, 0);
});
// ---- regression guard: pre-existing fields still computed ------------------
test('summarize — existing duration/event-name fields still correct on mixed input', () => {
const s = summarize(jsonl([
{ event: 'main-merge-gate', duration_ms: 100, ts: '2026-06-26T10:00:00Z' },
{ event: 'main-merge-approved', duration_ms: 200, ts: '2026-06-26T11:00:00Z' },
{ session_id: 'S1', tokens_input: 1000, cost_usd: 0.04, ts: '2026-06-26T12:00:00Z' },
]));
// Pre-existing behavior (regression guard)
assert.equal(s.total_events, 3);
assert.equal(s.events_with_duration, 2);
// Percentiles: durations [100,200] → floor(2·0.5)=floor(2·0.9)=idx 1 → both 200.
assert.equal(s.wall_time_ms_p50, 200);
assert.equal(s.wall_time_ms_p90, 200);
assert.equal(s.wall_time_ms_max, 200);
assert.deepEqual(s.unique_event_names, ['main-merge-approved', 'main-merge-gate']);
// Time range spans all ts-bearing lines (incl. the token-only record at 12:00).
assert.equal(s.oldest_event_iso, '2026-06-26T10:00:00.000Z');
assert.equal(s.newest_event_iso, '2026-06-26T12:00:00.000Z');
// New behavior coexists
assert.equal(s.total_tokens_input, 1000);
assert.equal(s.sessions_with_tokens, 1);
assert.ok(Math.abs(s.total_cost_usd - 0.04) < 1e-9);
});
test('summarize — malformed/empty lines tolerated alongside token records', () => {
const s = summarize([
'',
'not json',
JSON.stringify({ session_id: 'S1', tokens_input: 50, cost_usd: 0.001 }),
]);
assert.equal(s.total_events, 1);
assert.equal(s.sessions_with_tokens, 1);
assert.equal(s.total_tokens_input, 50);
});

294
tests/lib/claimed-order.test.mjs
View file

@ -1,294 +0,0 @@
// tests/lib/claimed-order.test.mjs
// The recorder that tells /trekbrief which coord-queue order it is running.
//
// It mirrors repo-mailbox's claim contract (coord root, mailbox = basename of the
// git toplevel, `_*` reserved, the claimed file's stem is the order id, a
// `<id>.claim` sidecar carries `claimed-at:`), and it must never guess: exactly
// one claimed order is an attribution, zero or several is `null` plus a reason.
// Every test builds its own coord dir and git repos; nothing reads the real
// queue. The module is loaded per test, so a missing module fails each test on
// its own instead of failing the file at link time.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync, readdirSync, statSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, dirname } from 'node:path';
import { spawnSync } from 'node:child_process';
const load = () => import('../../lib/stats/claimed-order.mjs');
// --- helpers ---------------------------------------------------------------
function fixture(files) {
const dir = mkdtempSync(join(tmpdir(), 'claimed-order-'));
for (const [rel, body] of Object.entries(files)) {
const p = join(dir, rel);
mkdirSync(dirname(p), { recursive: true });
writeFileSync(p, body);
}
return dir;
}
const cleanup = (...dirs) => { for (const d of dirs) rmSync(d, { recursive: true, force: true }); };
function orderFile({ id, to, quoted = false }) {
const lines = ['---', 'from: .claude'];
if (to !== undefined) lines.push(`to: ${to}`);
lines.push(`order-id: ${quoted ? `"${id}"` : id}`, `subject: order ${id}`, 'date: 2026-10-01T10:00:00Z', '---', '', `# ORDER ${id}`, '');
return lines.join('\n');
}
const claimFile = (at) => `claimed-at: ${at}\nclaimed-by-pid: 1\n`;
// A claimed order in <mailbox>, with its sidecar unless `claim: false`.
function claimed(mailbox, id, { at = '2026-10-04T10:00:00Z', claim = true, to = mailbox } = {}) {
const files = { [`${mailbox}/orders/claimed/${id}.md`]: orderFile({ id, to }) };
if (claim) files[`${mailbox}/orders/claimed/${id}.claim`] = claimFile(at);
return files;
}
function git(cwd, ...args) {
const r = spawnSync('git', ['-c', 'user.email=t@example.invalid', '-c', 'user.name=t', ...args], { cwd, encoding: 'utf8' });
assert.equal(r.status, 0, `git ${args.join(' ')}: ${r.stderr}`);
return r.stdout.trim();
}
// A git repo whose directory is named `name`, with one commit (a worktree needs a HEAD).
function repo(name) {
const parent = mkdtempSync(join(tmpdir(), 'claimed-order-git-'));
const dir = join(parent, name);
mkdirSync(dir);
git(dir, 'init', '-q');
writeFileSync(join(dir, 'README'), 'x\n');
git(dir, 'add', 'README');
git(dir, 'commit', '-q', '-m', 'init');
return { parent, dir };
}
function snapshot(dir) {
const out = [];
const walk = (d) => {
for (const n of readdirSync(d).sort()) {
const p = join(d, n);
const s = statSync(p);
out.push(`${p} ${s.mtimeMs}`);
if (s.isDirectory()) walk(p);
}
};
walk(dir);
return out.join('\n');
}
// --- findClaimedOrder ----------------------------------------------------------
test('findClaimedOrder: exactly one claimed order gives its order id and its claimed-at', async () => {
const { findClaimedOrder } = await load();
const coord = fixture(claimed('mbx', 'o-1', { at: '2026-10-04T17:10:51Z' }));
try {
assert.deepEqual(findClaimedOrder({ coordDir: coord, mailboxes: ['mbx'] }),
{ order_id: 'o-1', order_claimed_at: '2026-10-04T17:10:51Z', order_reason: null });
} finally { cleanup(coord); }
});
test('findClaimedOrder: no claimed/ directory gives null and none-claimed, never a guess', async () => {
const { findClaimedOrder } = await load();
const coord = fixture({ 'mbx/orders/o-1.md': orderFile({ id: 'o-1', to: 'mbx' }) });
try {
assert.deepEqual(findClaimedOrder({ coordDir: coord, mailboxes: ['mbx'] }),
{ order_id: null, order_claimed_at: null, order_reason: 'none-claimed' });
} finally { cleanup(coord); }
});
test('findClaimedOrder: an empty claimed/ gives null and none-claimed', async () => {
const { findClaimedOrder } = await load();
const coord = fixture({ 'mbx/orders/claimed/.keep': '' });
try {
assert.equal(findClaimedOrder({ coordDir: coord, mailboxes: ['mbx'] }).order_reason, 'none-claimed');
} finally { cleanup(coord); }
});
test('findClaimedOrder: two claimed orders give null and several-claimed:2 - the older one is never picked', async () => {
const { findClaimedOrder } = await load();
const coord = fixture({ ...claimed('mbx', 'o-1', { at: '2026-10-01T00:00:00Z' }), ...claimed('mbx', 'o-2') });
try {
assert.deepEqual(findClaimedOrder({ coordDir: coord, mailboxes: ['mbx'] }),
{ order_id: null, order_claimed_at: null, order_reason: 'several-claimed:2' });
} finally { cleanup(coord); }
});
test('findClaimedOrder: an unparseable claimed .md still counts - "several" is never undercounted', async () => {
const { findClaimedOrder } = await load();
const coord = fixture({ ...claimed('mbx', 'o-1'), 'mbx/orders/claimed/o-2.md': 'no frontmatter at all\n' });
try {
assert.equal(findClaimedOrder({ coordDir: coord, mailboxes: ['mbx'] }).order_reason, 'several-claimed:2');
} finally { cleanup(coord); }
});
test('findClaimedOrder: the .claim sidecar and other non-.md files are not orders', async () => {
const { findClaimedOrder } = await load();
const coord = fixture({ ...claimed('mbx', 'o-1'), 'mbx/orders/claimed/notes.txt': 'x\n' });
try {
assert.equal(findClaimedOrder({ coordDir: coord, mailboxes: ['mbx'] }).order_id, 'o-1');
} finally { cleanup(coord); }
});
test('findClaimedOrder: pending orders/ and archive/ are not claimed', async () => {
const { findClaimedOrder } = await load();
const coord = fixture({
'mbx/orders/o-1.md': orderFile({ id: 'o-1', to: 'mbx' }),
'mbx/orders/archive/o-2.md': orderFile({ id: 'o-2', to: 'mbx' }),
});
try {
assert.equal(findClaimedOrder({ coordDir: coord, mailboxes: ['mbx'] }).order_reason, 'none-claimed');
} finally { cleanup(coord); }
});
test("findClaimedOrder: another mailbox's claimed order is never ours", async () => {
const { findClaimedOrder } = await load();
const coord = fixture(claimed('other', 'o-1'));
try {
assert.equal(findClaimedOrder({ coordDir: coord, mailboxes: ['mbx'] }).order_reason, 'none-claimed');
} finally { cleanup(coord); }
});
test('findClaimedOrder: a missing coord dir gives null and coord-missing, without throwing', async () => {
const { findClaimedOrder } = await load();
const r = findClaimedOrder({ coordDir: join(tmpdir(), 'claimed-order-does-not-exist-'), mailboxes: ['mbx'] });
assert.deepEqual(r, { order_id: null, order_claimed_at: null, order_reason: 'coord-missing' });
});
test('findClaimedOrder: a claimed order without its .claim keeps the id, with no-claim-time', async () => {
const { findClaimedOrder } = await load();
const coord = fixture(claimed('mbx', 'o-1', { claim: false }));
try {
assert.deepEqual(findClaimedOrder({ coordDir: coord, mailboxes: ['mbx'] }),
{ order_id: 'o-1', order_claimed_at: null, order_reason: 'no-claim-time' });
} finally { cleanup(coord); }
});
test('findClaimedOrder: a to: that names another repo is to-mismatch; a missing to: is accepted', async () => {
const { findClaimedOrder } = await load();
const wrong = fixture(claimed('mbx', 'o-1', { to: 'elsewhere' }));
const none = fixture(claimed('mbx', 'o-2', { to: undefined }));
try {
assert.deepEqual(findClaimedOrder({ coordDir: wrong, mailboxes: ['mbx'] }),
{ order_id: null, order_claimed_at: null, order_reason: 'to-mismatch' });
assert.equal(findClaimedOrder({ coordDir: none, mailboxes: ['mbx'] }).order_id, 'o-2');
} finally { cleanup(wrong, none); }
});
test('findClaimedOrder: the id is the file stem, also when the frontmatter quotes order-id', async () => {
const { findClaimedOrder } = await load();
const coord = fixture({
'mbx/orders/claimed/o-1.md': orderFile({ id: 'o-1', to: 'mbx', quoted: true }),
'mbx/orders/claimed/o-1.claim': claimFile('2026-10-04T10:00:00Z'),
});
try {
assert.equal(findClaimedOrder({ coordDir: coord, mailboxes: ['mbx'] }).order_id, 'o-1');
} finally { cleanup(coord); }
});
test('findClaimedOrder: a _-prefixed mailbox is never read (the claim script reserves the namespace)', async () => {
const { findClaimedOrder } = await load();
const coord = fixture(claimed('_retired', 'o-1'));
try {
assert.equal(findClaimedOrder({ coordDir: coord, mailboxes: ['_retired'] }).order_id, null);
} finally { cleanup(coord); }
});
test('findClaimedOrder: it only reads - the coord dir is identical before and after', async () => {
const { findClaimedOrder } = await load();
const coord = fixture({ ...claimed('mbx', 'o-1'), ...claimed('other', 'o-2') });
try {
const before = snapshot(coord);
findClaimedOrder({ coordDir: coord, mailboxes: ['mbx', 'other'] });
findClaimedOrder({ coordDir: coord, mailboxes: ['mbx'] });
assert.equal(snapshot(coord), before);
} finally { cleanup(coord); }
});
test('findClaimedOrder: one claimed order across two mailbox names is found; one in each is several', async () => {
const { findClaimedOrder } = await load();
const one = fixture(claimed('main-repo', 'o-1'));
const two = fixture({ ...claimed('main-repo', 'o-1'), ...claimed('wt-name', 'o-2') });
try {
assert.equal(findClaimedOrder({ coordDir: one, mailboxes: ['wt-name', 'main-repo'] }).order_id, 'o-1');
assert.equal(findClaimedOrder({ coordDir: two, mailboxes: ['wt-name', 'main-repo'] }).order_reason, 'several-claimed:2');
} finally { cleanup(one, two); }
});
// --- resolveCoordDir -------------------------------------------------------------
test('resolveCoordDir: CLAUDE_COORD_DIR wins, else HOME/.claude/coord from the given env', async () => {
const { resolveCoordDir } = await load();
assert.equal(resolveCoordDir({ CLAUDE_COORD_DIR: '/x/coord', HOME: '/h' }), '/x/coord');
assert.equal(resolveCoordDir({ HOME: '/h' }), join('/h', '.claude', 'coord'));
assert.equal(resolveCoordDir({ CLAUDE_COORD_DIR: '', HOME: '/h' }), join('/h', '.claude', 'coord'));
});
// --- resolveMailboxes ------------------------------------------------------------
test('resolveMailboxes: a repo gives its basename; a worktree gives its own and the main repo name; outside git gives []', async () => {
const { resolveMailboxes } = await load();
const main = repo('main-repo');
const outside = mkdtempSync(join(tmpdir(), 'claimed-order-nogit-'));
try {
assert.deepEqual(resolveMailboxes(main.dir, { env: {} }), ['main-repo']);
const wt = join(main.parent, 'wt-name');
git(main.dir, 'worktree', 'add', '-q', '-b', 'wt-branch', wt);
assert.deepEqual([...resolveMailboxes(wt, { env: {} })].sort(), ['main-repo', 'wt-name']);
assert.deepEqual(resolveMailboxes(outside, { env: {} }), []);
} finally { cleanup(main.parent, outside); }
});
test('resolveMailboxes: an inherited GIT_DIR pointing at another repo is ignored', async () => {
const { resolveMailboxes } = await load();
const mine = repo('mine');
const theirs = repo('theirs');
try {
const env = { GIT_DIR: join(theirs.dir, '.git'), GIT_WORK_TREE: theirs.dir };
assert.deepEqual(resolveMailboxes(mine.dir, { env }), ['mine']);
} finally { cleanup(mine.parent, theirs.parent); }
});
// --- attributeOrder --------------------------------------------------------------
test('attributeOrder: outside git the mailbox is unresolved - null, never a guess', async () => {
const { attributeOrder } = await load();
const outside = mkdtempSync(join(tmpdir(), 'claimed-order-nogit-'));
const coord = fixture(claimed('mbx', 'o-1'));
try {
assert.deepEqual(attributeOrder({ cwd: outside, env: { CLAUDE_COORD_DIR: coord, HOME: outside } }),
{ order_id: null, order_claimed_at: null, order_reason: 'mailbox-unresolved' });
} finally { cleanup(outside, coord); }
});
test('attributeOrder: finds the claimed order of the cwd repo; a projectDir outside cwd cannot steer it elsewhere', async () => {
const { attributeOrder } = await load();
const a = repo('repo-a');
const b = repo('repo-b');
const coord = fixture(claimed('repo-b', 'o-1'));
try {
const env = { CLAUDE_COORD_DIR: coord, HOME: a.parent };
// Positive control: the fixture is findable from repo-b itself.
assert.equal(attributeOrder({ cwd: b.dir, env }).order_id, 'o-1');
// From repo-a, a project_dir that walks out of cwd into repo-b is ignored.
assert.equal(attributeOrder({ cwd: a.dir, env, projectDir: b.dir }).order_reason, 'none-claimed');
assert.equal(attributeOrder({ cwd: a.dir, env, projectDir: '../../' + b.parent.split('/').pop() + '/repo-b' }).order_reason, 'none-claimed');
} finally { cleanup(a.parent, b.parent, coord); }
});
test('attributeOrder: a projectDir inside cwd moves the lookup to the repo that holds it', async () => {
const { attributeOrder } = await load();
const m = repo('mbx');
const coord = fixture(claimed('mbx', 'o-1'));
try {
mkdirSync(join(m.dir, '.claude', 'projects', 'x'), { recursive: true });
const env = { CLAUDE_COORD_DIR: coord, HOME: m.parent };
// Control: cwd (the repo's parent) is outside git, so without projectDir there is no mailbox.
assert.equal(attributeOrder({ cwd: m.parent, env }).order_reason, 'mailbox-unresolved');
assert.deepEqual(attributeOrder({ cwd: m.parent, env, projectDir: 'mbx/.claude/projects/x' }),
{ order_id: 'o-1', order_claimed_at: '2026-10-04T10:00:00Z', order_reason: null });
} finally { cleanup(m.parent, coord); }
});

217
tests/lib/doc-consistency.test.mjs
View file

@ -23,7 +23,6 @@ import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { parseDocument } from '../../lib/util/frontmatter.mjs';
import { resolveProfile, loadProfile } from '../../lib/profiles/resolver.mjs';
import { TREKRESEARCH_ALLOWED } from '../../lib/exporters/field-allowlist.mjs';
const HERE = dirname(fileURLToPath(import.meta.url));
const ROOT = join(HERE, '..', '..');
@ -338,9 +337,7 @@ test('operations.md custom-profile prose matches findProfilePath resolution orde
// → {"ok":true,"gates":true|false}. It has no state machine (R9).
// The old prose (operations.md:15 / command-modes.md gates rows) (1) fabricated a
// state machine `idle → approved → executing → merge-pending → main-merged`, and
// later named the removed one (gates_on, auto_running, paused_for_gate); (2) claimed event-emit.mjs "records
// each transition" — it emits 3 named lifecycle events and is decoupled from the
// (pure, no-I/O) autonomy-gate; (3) advertised `--gates {open|closed|adaptive}` —
// later named the removed one (gates_on, auto_running, paused_for_gate); (3) advertised `--gates {open|closed|adaptive}` —
// there is no strictness policy and no `adaptive` value: the flag is boolean, and
// the CLI shim + all 4 command docs take `--gates [true|false|closed|open]`
// (bare, true, closed = on; false, open = off; fix round 4.5a, #1).
@ -362,12 +359,6 @@ test('operations.md + command-modes.md describe the real autonomy-gate, not a fa
'docs/operations.md must describe lib/util/autonomy-gate.mjs as the --gates flag parser',
);
// F2 — event-emit does NOT record each autonomy-gate transition
assert.ok(
!ops.includes('records each transition'),
'docs/operations.md: event-emit records 3 named lifecycle events, not "each transition" of the (decoupled) autonomy-gate; fix the prose',
);
// F3 — --gates is boolean; there is no open/closed/adaptive strictness enum
const GATES_ENUM_FORMS = ['{open|closed|adaptive}', '{open\\|closed\\|adaptive}'];
for (const [label, txt] of [['operations.md', ops], ['command-modes.md', modes]]) {
@ -917,31 +908,6 @@ test('S15: profile tables encode each built-in yaml phase_models exactly', () =>
}
});
// STRUCTURAL pin: the exporter allowlist and the authoring fixture must agree.
// `engine` was emitted, documented in prose, and still dropped at the export
// boundary because nothing tied the two together. Derives one side from the
// frozen Set, so it survives rewording of the fixture row.
test('S74: every TREKRESEARCH_ALLOWED name is declared in the jsonl-schemas fixture row', () => {
const row = read('tests/fixtures/jsonl-schemas.md')
.split('\n')
.find((l) => l.startsWith('| trekresearch-stats '));
assert.ok(row, 'jsonl-schemas.md is missing the `trekresearch-stats` row');
// Columns: '' | schema_id | fields | writer_path | line_ref | v4.1 additive | PII | ''
const cells = row.split('|').map((c) => c.trim());
// Both columns are required: profile/profile_source/parallel_agents live in
// the `v4.1 additive` column only, so checking `fields` alone fails at once.
const declared = new Set(
[cells[2], cells[5]].flatMap((c) => c.split(',').map((f) => f.trim())).filter(Boolean),
);
for (const name of TREKRESEARCH_ALLOWED) {
assert.ok(
declared.has(name),
`\`${name}\` is allowlisted in field-allowlist.mjs but absent from the fixture row's `
+ '`fields` + `v4.1 additive` columns — fix the SOURCE, not this pin',
);
}
});
// --- S34 (V30) — economy is self-declared experimental until the cross-tier
// Jaccard floor (0.55) is empirically calibrated (Step-17 calibration deferred
// to v4.2). The status must be visible in BOTH the profile data
@ -1399,26 +1365,6 @@ test('S32: trekbrief Phase 3 delegates Q&A turn-taking to AskUserQuestion (V01
);
});
// ── S33 — documentation consolidation: record the considered-and-kept decisions ──
//
// D2 (V32 rationale) — observability.md documents why direct export rather
// than a native collector (preserves the S21 in-process guards).
// Fix the SOURCE a pin guards, not the test.
test('S33: docs/observability.md documents the direct-export-vs-collector rationale (D2)', () => {
const t = read('docs/observability.md');
assert.ok(
/direct[- ]export/i.test(t) && /collector/i.test(t),
'observability.md must document why Voyage exports directly rather than via a native collector (D2)',
);
// The rationale must anchor on the in-process security boundary it preserves
// (the S21 hardening), not read as an unjustified re-host.
assert.ok(
/allowlist/i.test(t) && /(SSRF|CWE-918|CWE-212)/.test(t),
'the direct-export rationale must cite the in-process guards it preserves (field allowlist + SSRF/path)',
);
});
// ── S38 — fan-out hedge harmonization + relief-assertion forward-guard (SC d) ──
//
// The brief's SC(d) premise — an unhedged "fan-out relieves main context" string
@ -1560,10 +1506,6 @@ test('v5.9: no commands/*.md frontmatter carries a model: key (session inheritan
// version did or what was measured then, and rewriting them is the same defect
// class as bumping a version string inside a measurement doc.
//
// `gemini_used` is deliberately NOT banned. It is a vestigial trekresearch stats
// field, pinned to false by the drop and kept so the observability export schema
// does not break for existing consumers (see TREKRESEARCH_ALLOWED).
//
// BLIND SPOT, stated rather than hidden: THIS file cannot be in the live-surface
// list, because the S95 block below is necessarily full of the word. So the two
// v5.1.1 assertion messages above (which describe the high-effort rules) are not
@ -1579,7 +1521,6 @@ const GEMINI_LIVE_SURFACES = [
'docs/command-modes.md',
'templates/research-brief-template.md',
'templates/trekbrief-template.md',
'tests/fixtures/jsonl-schemas.md',
...listMd('commands').map((f) => `commands/${f}`),
...listMd('agents').map((f) => `agents/${f}`),
];
@ -1601,10 +1542,7 @@ test('S95: agents/gemini-bridge.md is removed from the agent inventory', () => {
test('S95: no live pipeline surface still names the gemini-bridge agent', () => {
const offenders = [];
for (const rel of GEMINI_LIVE_SURFACES) {
// Everything gemini EXCEPT the vestigial gemini_used stats field.
const hits = (read(rel).match(/gemini[\w-]*/gi) || []).filter(
(m) => m.toLowerCase() !== 'gemini_used',
);
const hits = read(rel).match(/gemini[\w-]*/gi) || [];
if (hits.length) offenders.push(`${rel} (${[...new Set(hits)].join(', ')})`);
}
assert.deepEqual(
@ -1635,13 +1573,6 @@ test('S95: trekplan high-effort Pass 2 degrades explicitly instead of failing',
);
});
test('S95: gemini_used survives as a vestigial stats field (export schema unbroken)', () => {
assert.ok(
TREKRESEARCH_ALLOWED.has('gemini_used'),
'gemini_used stays in the trekresearch export allowlist — the agent is dropped, the field is pinned false, and removing it would break existing consumers',
);
});
// End-state defect D-07: since v5.9.0 no command pins `model:`, so the orchestrator follows
// the session model, and sub-agent models are resolved at the spawn sites (brief signal >
// profile > agent frontmatter). No user-facing doc may claim otherwise. Fix the SOURCE.
@ -1862,94 +1793,6 @@ test('model aliases: profiles.md states the current targets, defaults and sessio
}
});
// --- the order a /trekbrief run records (yardstick use, source C) ---------------
// Five surfaces for order_id / order_claimed_at / order_reason: (1) record format
// = the writer, lib/stats/stats-append.mjs (pinned behaviourally in
// tests/lib/stats-landing.test.mjs); (2) measured-field prose = /trekbrief Phase 7;
// (3) field allowlist = denied, said in the allowlist header; (4) jsonl schemas =
// the trekbrief-stats fixture row + EXPORT_DENYLIST; (5) docs = docs/architecture.md.
const ORDER_FIELDS = ['order_id', 'order_claimed_at', 'order_reason'];
function mdSection(text, heading) {
const start = text.indexOf(`\n## ${heading}`);
assert.ok(start >= 0, `missing "## ${heading}"`);
const next = text.indexOf('\n## ', start + 1);
return next < 0 ? text.slice(start) : text.slice(start, next);
}
function trekbriefRow() {
const row = read('tests/fixtures/jsonl-schemas.md').split('\n').find((l) => l.startsWith('| trekbrief-stats '));
assert.ok(row, 'jsonl-schemas.md is missing the `trekbrief-stats` row');
return row.split('|').map((c) => c.trim());
}
test('order fields: /trekbrief Phase 7 names them and says the writer adds them', () => {
const sec = mdSection(read('commands/trekbrief.md'), 'Phase 7');
for (const f of ORDER_FIELDS) assert.ok(sec.includes(`\`${f}\``), `Phase 7 must name \`${f}\``);
assert.match(sec, /writer adds/i, 'the model never types the order id - the prose must say who adds it');
});
test('order fields: the trekbrief-stats fixture row lists them in `fields`, and EXPORT_DENYLIST names them', () => {
const fields = new Set(trekbriefRow()[2].split(',').map((f) => f.trim()));
for (const f of ORDER_FIELDS) assert.ok(fields.has(f), `\`${f}\` missing from the trekbrief-stats \`fields\` column`);
const text = read('tests/fixtures/jsonl-schemas.md');
const deny = text.slice(text.indexOf('**EXPORT_DENYLIST**'));
const block = deny.slice(deny.indexOf('```') + 3, deny.indexOf('```', deny.indexOf('```') + 3));
for (const f of ORDER_FIELDS) assert.ok(block.split(',').map((x) => x.trim()).includes(f), `\`${f}\` missing from EXPORT_DENYLIST`);
});
test('order fields: the trekbrief-stats row\'s line_ref still points at the Phase 7 stats-append line', () => {
const ref = trekbriefRow()[4];
const m = ref.match(/^trekbrief\.md:(\d+)-(\d+)$/);
assert.ok(m, `line_ref "${ref}" is not trekbrief.md:A-B`);
const lines = read('commands/trekbrief.md').split('\n');
const at = lines.findIndex((l) => /^\s*node \$\{CLAUDE_PLUGIN_ROOT\}\/lib\/stats\/stats-append\.mjs trekbrief\b/.test(l)) + 1;
assert.ok(at > 0, 'the Phase 7 stats-append line is gone');
assert.ok(at >= Number(m[1]) && at <= Number(m[2]), `line_ref ${ref} does not contain the stats-append line (line ${at})`);
});
/** The jsonl-schemas fixture row whose schema_id cell is `id`, split into cells. */
function schemaRow(id) {
const row = read('tests/fixtures/jsonl-schemas.md').split('\n').find((l) => l.startsWith(`| ${id} |`));
assert.ok(row, `jsonl-schemas.md is missing the \`${id}\` row`);
return row.split('|').map((c) => c.trim());
}
// The Phase 9 line_ref was measured, then more prose landed above Phase 9 and
// nothing re-measured it. The same drift hit the other command stats rows.
// Each row is pinned to the span it names: its phase heading to the closing
// fence of its stats-append stdin block. The trekreview anchor is the full
// "## Phase 8 — Validate output" so the Phase 8.5 / 8.7 headings never match.
// The trekresearch anchor is "### Stats tracking", the one such heading in
// commands/trekresearch.md.
const STATS_ROWS = [
{ id: 'trekbrief-stats', cmd: 'trekbrief', heading: /^## Phase 7\b(?!\.)/ },
{ id: 'trekresearch-stats', cmd: 'trekresearch', heading: /^### Stats tracking$/ },
{ id: 'trekplan-stats', cmd: 'trekplan', heading: /^## Phase 12\b(?!\.)/ },
{ id: 'trekreview-stats', cmd: 'trekreview', heading: /^## Phase 8 — Validate output/ },
{ id: 'trekexecute-stats (Phase 9 record)', cmd: 'trekexecute', heading: /^## Phase 9\b(?!\.)/ },
];
for (const { id, cmd, heading: headingRe } of STATS_ROWS) {
test(`the ${id} row's line_ref spans its phase heading to the end of its stats-append stdin block`, () => {
const ref = schemaRow(id)[4];
const m = ref.match(new RegExp(`^${cmd}\\.md:(\\d+)-(\\d+)$`));
assert.ok(m, `line_ref "${ref}" is not ${cmd}.md:A-B`);
const lines = read(`commands/${cmd}.md`).split('\n');
const headings = lines.map((l, i) => (headingRe.test(l) ? i : -1)).filter((i) => i >= 0);
assert.equal(headings.length, 1, `exactly one heading in commands/${cmd}.md matches ${headingRe}`);
const heading = headings[0] + 1;
const appendRe = new RegExp(`^\\s*node \\$\\{CLAUDE_PLUGIN_ROOT\\}/lib/stats/stats-append\\.mjs ${cmd}\\b`);
const append = lines.findIndex((l, i) => i >= heading && appendRe.test(l)) + 1;
const nextPhase = lines.findIndex((l, i) => i >= heading && /^## Phase /.test(l)) + 1;
assert.ok(append > heading, `a stats-append line follows the heading in commands/${cmd}.md`);
assert.ok(nextPhase === 0 || append < nextPhase, `the stats-append line sits inside the phase (line ${append}, next phase at ${nextPhase})`);
const fence = lines.findIndex((l, i) => i >= append && l.trim() === '```') + 1;
assert.ok(fence > append, 'the stats-append block has a closing fence');
assert.equal(ref, `${cmd}.md:${heading}-${fence}`);
});
}
// A `## Brief Quality` section belongs to the exhausted and force-stopped
// branches only; the passed-gate branch must not tell the model to write one.
test('trekbrief: the passed-gate branch names no `## Brief Quality`, the exhausted branch still does', () => {
@ -1964,62 +1807,6 @@ test('trekbrief: the passed-gate branch names no `## Brief Quality`, the exhaust
assert.ok(text.slice(exStart, exEnd).includes('## Brief Quality'), 'the exhausted branch must still name `## Brief Quality`');
});
test('the autonomy-events row\'s line_ref spans emit() in event-emit.mjs', () => {
const ref = schemaRow('trekexecute-stats (autonomy events)')[4];
const m = ref.match(/^event-emit\.mjs:(\d+)-(\d+)$/);
assert.ok(m, `line_ref "${ref}" is not event-emit.mjs:A-B`);
const lines = read('lib/stats/event-emit.mjs').split('\n');
const start = lines.findIndex((l) => l.startsWith('export function emit(')) + 1;
const end = lines.findIndex((l, i) => i >= start && l === '}') + 1;
assert.ok(start > 0 && end > start, 'emit() and its closing brace');
assert.equal(ref, `event-emit.mjs:${start}-${end}`);
});
test('order fields: docs/architecture.md Stats names them and the order-id join', () => {
const text = read('docs/architecture.md');
const stats = text.slice(text.indexOf('\nStats:'), text.indexOf('\n## Terminology'));
for (const f of ORDER_FIELDS) assert.ok(stats.includes(`\`${f}\``), `docs/architecture.md Stats must name \`${f}\``);
assert.match(stats, /yardstick/);
});
test('order fields: the field-allowlist header says they are denied, and TREKBRIEF_ALLOWED leaves them out', () => {
const text = read('lib/exporters/field-allowlist.mjs');
const header = text.slice(0, text.indexOf('const TREKBRIEF_ALLOWED'));
for (const f of ORDER_FIELDS) assert.ok(header.includes(f), `the allowlist header must name ${f} among the dropped fields`);
const body = text.slice(text.indexOf('const TREKBRIEF_ALLOWED'), text.indexOf(']));', text.indexOf('const TREKBRIEF_ALLOWED')));
for (const f of ORDER_FIELDS) assert.ok(!body.includes(`'${f}'`), `${f} must not be allowlisted for export`);
});
test('yardstick header: use joins on the order id, says where a no comes from, and names the false-positive class', () => {
const text = read('scripts/yardstick.mjs');
const header = text.slice(0, text.indexOf('\nimport '));
assert.match(header, /order id/);
assert.match(header, /arbeidsform: ordre/);
assert.match(header, /false-positive/);
assert.match(header, /USE_COUNTING_SINCE/);
assert.doesNotMatch(header, /slug equals the order's/, 'the slug join is gone');
});
test('/trekbrief --approve writes the trekbrief record: Phase 1 and Phase 4h run Phase 7 with mode "approve"', () => {
const text = read('commands/trekbrief.md');
const item4 = text.slice(text.indexOf('4. `--approve <project-dir | brief-file>`'), text.indexOf('In every other mode'));
assert.match(item4, /Phase 7/, 'approve mode must reach Phase 7 (PM 04.10: the approval leaves the trekbrief record)');
const tail = text.slice(text.indexOf('In `--approve` mode'), text.indexOf('## Phase 5'));
assert.match(tail, /Phase 7/);
const phase7 = mdSection(text, 'Phase 7');
assert.match(phase7, /"mode": "\{default \| quick \| approve\}"/);
assert.match(phase7, /"mode": "approve"/, 'Phase 7 must fix the approve-mode values');
assert.match(text, /Mode: \{default \| quick \| approve\}/, 'the Phase 1 report lists approve');
});
test('/trekbrief --approve: CLAUDE.md, README.md and docs/command-modes.md say the approval writes the trekbrief record', () => {
for (const rel of ['CLAUDE.md', 'README.md', 'docs/command-modes.md']) {
const line = read(rel).split('\n').find((l) => l.includes('--approve <'));
assert.ok(line, `${rel} has no --approve row`);
assert.match(line, /trekbrief (stats )?record/, `${rel}: the --approve row must say it writes the trekbrief record`);
}
});
// --- effort floor: no live doc claims retrieval agents run at medium ------------
// Operator rule (2026-10-05): no agent is pinned below `high`. The model DEFAULT
// sentence ("Opus 5.5 ... default reasoning effort `medium`") stays true and is not a pin.

4
tests/lib/executor-prose-pins.test.mjs
View file

@ -222,8 +222,8 @@ test('P4: the run stops only on the closed list of four, and the final report sh
}
assert.match(list, /closed/i);
// Phase 8's report template is a fence that holds its own "## " heading, so
// the phase is read up to Phase 9, not up to the next heading.
const report = text.slice(text.indexOf('## Phase 8 — Final report'), text.indexOf('## Phase 9'));
// the phase is read up to Hard rules, not up to the next heading.
const report = text.slice(text.indexOf('## Phase 8 — Final report'), text.indexOf('## Hard rules'));
assert.match(report, /### Stops/);
assert.match(report, /### Decisions taken/);
});

49
tests/lib/gold-eval.test.mjs
View file

@ -1,49 +0,0 @@
// tests/lib/gold-eval.test.mjs
// SKAL-1·4b — the offline gold-scored output eval (the scoring RUN).
//
// This is the third test-census category (see lib/util/test-census.mjs):
// neither a behavior unit test nor a doc-consistency pin, but a SCORING RUN —
// it feeds a committed agent-run fixture through the deterministic coordinator
// contract (4a) and scores the result against the golden corpus at
// (file, rule_key) granularity. Offline: committed reviewer payloads, no live
// agent spawn, no LLM, no network. The all-agree foundation; the
// LLM-in-the-loop eval is the separate 4c tier.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { readFileSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { runContract } from '../../lib/review/coordinator-contract.mjs';
import { scoreFindings, scoreVerdict } from '../../lib/review/gold-scorer.mjs';
const HERE = dirname(fileURLToPath(import.meta.url));
const ROOT = join(HERE, '..', '..');
const gold = JSON.parse(readFileSync(join(ROOT, 'tests/fixtures/bakeoff-rich/gold.json'), 'utf-8'));
const runPerfect = JSON.parse(
readFileSync(join(ROOT, 'tests/fixtures/bakeoff-rich/runs/run-perfect.json'), 'utf-8'),
);
test('gold-eval — run-perfect reproduces gold at (file,rule_key): precision/recall/f1 = 1.0', () => {
const result = runContract(runPerfect);
const score = scoreFindings(result.findings, gold.findings);
assert.equal(score.precision, 1, `precision ${score.precision}; spurious=${JSON.stringify(score.spurious)}`);
assert.equal(score.recall, 1, `recall ${score.recall}; missed=${JSON.stringify(score.missed)}`);
assert.equal(score.f1, 1);
assert.equal(score.tp, gold.findings.length);
});
test('gold-eval — run-perfect coordinator verdict matches gold expected_verdict (BLOCK)', () => {
const result = runContract(runPerfect);
assert.equal(result.verdict, gold.expected_verdict);
assert.ok(scoreVerdict(result.verdict, gold.expected_verdict));
});
test('gold-eval — run-perfect drops nothing (no suppressed, no schema-skipped payloads)', () => {
// A clean run is the regression guard: any future contract change that
// silently suppresses or skips one of the 5 seeded findings breaks here.
const result = runContract(runPerfect);
assert.equal(result.skipped.length, 0, `skipped payloads: ${JSON.stringify(result.skipped)}`);
assert.equal(result.suppressed.length, 0, `suppressed findings: ${result.suppressed.length}`);
assert.equal(result.findings.length, gold.findings.length);
});

126
tests/lib/gold-scorer.test.mjs
View file

@ -1,126 +0,0 @@
// tests/lib/gold-scorer.test.mjs
// SKAL-1·4b — behavior test for the gold scorer.
//
// scoreFindings compares a recorded agent-run's findings against a golden
// corpus record at (file, rule_key) granularity (line + severity deliberately
// ignored). This pins the precision/recall/f1 math AND the discriminating
// paths (false negatives + false positives), not merely the all-match case —
// a scorer that always returned 1.0 would pass an all-match-only test.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { scoreFindings, scoreVerdict } from '../../lib/review/gold-scorer.mjs';
// A minimal gold set: two distinct (file, rule_key) pairs.
const GOLD = [
{ file: 'a.mjs', line: 10, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' },
{ file: 'b.mjs', line: 0, rule_key: 'MISSING_TEST', severity: 'MAJOR' },
];
test('scoreFindings — perfect match scores precision/recall/f1 = 1', () => {
// Same pairs, different line/severity (must be ignored at (file,rule_key) granularity).
const run = [
{ file: 'a.mjs', line: 99, rule_key: 'SECURITY_INJECTION', severity: 'MINOR' },
{ file: 'b.mjs', line: 7, rule_key: 'MISSING_TEST', severity: 'BLOCKER' },
];
const s = scoreFindings(run, GOLD);
assert.equal(s.tp, 2);
assert.equal(s.fp, 0);
assert.equal(s.fn, 0);
assert.equal(s.precision, 1);
assert.equal(s.recall, 1);
assert.equal(s.f1, 1);
});
test('scoreFindings — a missed gold pair is a false negative (recall < 1)', () => {
const run = [{ file: 'a.mjs', line: 10, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' }];
const s = scoreFindings(run, GOLD);
assert.equal(s.tp, 1);
assert.equal(s.fp, 0);
assert.equal(s.fn, 1);
assert.equal(s.precision, 1);
assert.equal(s.recall, 0.5);
assert.deepEqual(s.missed, ['b.mjs MISSING_TEST']);
});
test('scoreFindings — a spurious pair is a false positive (precision < 1)', () => {
const run = [
{ file: 'a.mjs', line: 10, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' },
{ file: 'b.mjs', line: 0, rule_key: 'MISSING_TEST', severity: 'MAJOR' },
{ file: 'c.mjs', line: 3, rule_key: 'PLACEHOLDER_IN_CODE', severity: 'MAJOR' },
];
const s = scoreFindings(run, GOLD);
assert.equal(s.tp, 2);
assert.equal(s.fp, 1);
assert.equal(s.fn, 0);
assert.equal(s.recall, 1);
assert.equal(s.precision, 2 / 3);
assert.deepEqual(s.spurious, ['c.mjs PLACEHOLDER_IN_CODE']);
});
test('scoreFindings — mixed FN + FP', () => {
const run = [
{ file: 'a.mjs', line: 10, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' }, // match
{ file: 'c.mjs', line: 3, rule_key: 'PLACEHOLDER_IN_CODE', severity: 'MAJOR' }, // spurious
];
const s = scoreFindings(run, GOLD);
assert.equal(s.tp, 1);
assert.equal(s.fp, 1);
assert.equal(s.fn, 1);
assert.equal(s.precision, 0.5);
assert.equal(s.recall, 0.5);
assert.equal(s.f1, 0.5);
});
test('scoreFindings — duplicate (file,rule_key) pairs collapse (set semantics)', () => {
const run = [
{ file: 'a.mjs', line: 10, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' },
{ file: 'a.mjs', line: 22, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' }, // same pair
{ file: 'b.mjs', line: 0, rule_key: 'MISSING_TEST', severity: 'MAJOR' },
];
const s = scoreFindings(run, GOLD);
assert.equal(s.tp, 2);
assert.equal(s.fp, 0);
assert.equal(s.fn, 0);
});
test('scoreFindings — empty run: recall 0 (nothing found), precision vacuously 1, f1 0', () => {
const s = scoreFindings([], GOLD);
assert.equal(s.tp, 0);
assert.equal(s.fp, 0);
assert.equal(s.fn, 2);
assert.equal(s.recall, 0);
assert.equal(s.precision, 1);
assert.equal(s.f1, 0);
});
test('scoreFindings — empty gold: precision 0 (all spurious), recall vacuously 1, f1 0', () => {
const run = [{ file: 'a.mjs', line: 10, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER' }];
const s = scoreFindings(run, []);
assert.equal(s.tp, 0);
assert.equal(s.fp, 1);
assert.equal(s.fn, 0);
assert.equal(s.precision, 0);
assert.equal(s.recall, 1);
assert.equal(s.f1, 0);
});
test('scoreFindings — both empty: precision/recall/f1 = 1 (matched nothing perfectly)', () => {
const s = scoreFindings([], []);
assert.equal(s.precision, 1);
assert.equal(s.recall, 1);
assert.equal(s.f1, 1);
});
test('scoreFindings — tolerates null/undefined finding arrays', () => {
const s = scoreFindings(null, undefined);
assert.equal(s.tp, 0);
assert.equal(s.fp, 0);
assert.equal(s.fn, 0);
});
test('scoreVerdict — exact verdict match is true, mismatch is false', () => {
assert.equal(scoreVerdict('BLOCK', 'BLOCK'), true);
assert.equal(scoreVerdict('WARN', 'BLOCK'), false);
assert.equal(scoreVerdict('ALLOW', 'ALLOW'), true);
});

6
tests/lib/living-paths.test.mjs
View file

@ -109,12 +109,6 @@ const EXEMPT = new Map([
['docs/HANDOVER-CONTRACTS.md docs/specs/api.md', TARGET],
['templates/trekreview-template.md lib/foo.mjs', TARGET],
['docs/release-runbook.md scripts/release-plugin.mjs', "the catalog repo's script"],
['docs/T1-cc26-delegated-orchestration.md scripts/q3-cache-prefix-experiment.mjs', RB],
['docs/T1-cc26-delegated-orchestration.md docs/cc-upgrade-2.1.181-decision-matrix.md', RB],
['docs/T1-cc26-delegated-orchestration.md docs/subagent-delegation-audit.md', RB],
['docs/T1-cc26-delegated-orchestration.md agents/planning-orchestrator.md', RB],
['docs/T1-cc26-delegated-orchestration.md docs/T1-synthesis-poc-results.md', RB],
['docs/T1-cc26-delegated-orchestration.md agents/synthesis-agent.md', RB],
['docs/deep-research-engine-research.md docs/cc-upgrade-2.1.181-decision-matrix.md', RB],
]);

2
tests/lib/no-claude-code-launch.test.mjs
View file

@ -51,11 +51,9 @@ const PATTERNS = [
const EXEMPT = new Map([
[SELF, ['self', 17, "the gate's own pattern table and known-positive fixture"]],
['CHANGELOG.md', ['c', 15, 'release history; never rewritten']],
['docs/T1-cc26-delegated-orchestration.md', ['c', 4, 'measurement record (T1/CC-26)']],
['docs/deep-research-engine-research.md', ['c', 6, 'research record (deep-research engine)']],
['docs/operations.md', ['c', 1, 'Path A decision record: why cache-first was rejected']],
['docs/spike-pretooluse-subagent-reach.md', ['c', 2, 'spike record (PreToolUse reach)']],
['scripts/synthesis-measure.mjs', ['c', 2, 'measurement script; says why a child-session instrument was not used, launches nothing']],
['tests/fixtures/research-deep-research-adapted.md', ['c', 1, 'recorded research fixture']],
['docs/proevesett.md', ['operator-instrument', 3, "the operator's own eval run through Anthropic's `claude plugin eval`; Voyage starts nothing - kept by operator decision 2026-10-02"]],
['.gitignore', ['operator-instrument', 1, 'ignores the output of that same operator-run eval']],

224
tests/lib/profile-stats-fields.test.mjs
View file

@ -1,224 +0,0 @@
// tests/lib/profile-stats-fields.test.mjs
// SC #11 contract-test per brief design — kombinasjonen av:
// (a) fixture-records valideres som JSONL-contracts AND
// (b) command prose promises no profile key its stats record does not write
// er den brief-designede gating-mekanismen. Faktisk runtime-emission av
// feltene er LLM-prose-driven og ikke testbart i node:test alene.
// (b) was turned around in fix round 4-5a (#46): it used to require the prose
// to name profile_source and friends, which no stats heredoc writes.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { readFileSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
const __dirname = dirname(fileURLToPath(import.meta.url));
const REPO_ROOT = join(__dirname, '..', '..');
const PROFILE_FIELDS = [
'profile',
'phase_models',
'parallel_agents',
'external_research_enabled',
'profile_source',
'model_used',
'phase_models_resolved',
];
const VALID_PROFILE_SOURCES = new Set(['flag', 'env', 'default', 'inheritance']);
const COMMAND_FILES = [
'trekbrief.md',
'trekresearch.md',
'trekplan.md',
'trekexecute.md',
'trekreview.md',
'trekcontinue.md',
];
// (a) Fixture validates as JSONL contracts
test('SC #11(a): tests/fixtures/stats-with-profile.jsonl parses as JSONL', () => {
const text = readFileSync(join(REPO_ROOT, 'tests', 'fixtures', 'stats-with-profile.jsonl'), 'utf-8');
const lines = text.trim().split('\n').filter(Boolean);
assert.equal(lines.length, 5, `expected 5 simulated stats records, got ${lines.length}`);
for (const line of lines) {
const record = JSON.parse(line); // throws if malformed
assert.equal(typeof record, 'object');
assert.ok(record.ts, 'record missing ts');
}
});
test('SC #11(a): every fixture record contains profile + profile_source', () => {
const text = readFileSync(join(REPO_ROOT, 'tests', 'fixtures', 'stats-with-profile.jsonl'), 'utf-8');
const records = text.trim().split('\n').filter(Boolean).map(l => JSON.parse(l));
for (const r of records) {
assert.ok('profile' in r, `record missing profile: ${JSON.stringify(r)}`);
assert.ok('profile_source' in r, `record missing profile_source: ${JSON.stringify(r)}`);
}
});
test('SC #11(a): profile_source values are in {flag, env, default, inheritance}', () => {
const text = readFileSync(join(REPO_ROOT, 'tests', 'fixtures', 'stats-with-profile.jsonl'), 'utf-8');
const records = text.trim().split('\n').filter(Boolean).map(l => JSON.parse(l));
for (const r of records) {
assert.ok(VALID_PROFILE_SOURCES.has(r.profile_source),
`profile_source "${r.profile_source}" not in valid set`);
}
});
test('SC #11(a): fixture coverage — all 4 profile_source values represented', () => {
const text = readFileSync(join(REPO_ROOT, 'tests', 'fixtures', 'stats-with-profile.jsonl'), 'utf-8');
const records = text.trim().split('\n').filter(Boolean).map(l => JSON.parse(l));
const seen = new Set(records.map(r => r.profile_source));
for (const expected of VALID_PROFILE_SOURCES) {
assert.ok(seen.has(expected),
`fixture missing profile_source value: ${expected}; seen: ${[...seen].join(', ')}`);
}
});
// (b) Command prose promises no profile key its stats record does not write
// (false-confidence kompensasjon per plan-critic Major 4). A paragraph outside
// a code fence that mentions stats may name a PROFILE_FIELDS key in backticks
// only when the command's own `stats-append.mjs <kind> <<'JSON'` heredoc
// writes it.
/** The keys of every `stats-append.mjs <kind> <<'JSON'` … `JSON` heredoc in `text`. */
function statsHeredocs(text) {
const out = [];
for (const m of text.matchAll(/stats-append\.mjs (\S+) <<'JSON'\n([\s\S]*?)\nJSON\s*$/gm)) {
out.push({ kind: m[1], keys: new Set([...m[2].matchAll(/"([A-Za-z_][A-Za-z0-9_]*)"\s*:/g)].map((k) => k[1])) });
}
return out;
}
/** Backticked PROFILE_FIELDS keys in stats paragraphs (outside fences) that `written` lacks; sorted. */
function unwrittenStatsKeys(text, written) {
const paras = [];
let cur = [];
let fenced = false;
for (const l of text.split('\n')) {
if (/^\s{0,3}(`{3,}|~{3,})/.test(l)) { fenced = !fenced; paras.push(cur); cur = []; continue; }
if (fenced) continue;
if (l.trim() === '') { paras.push(cur); cur = []; } else cur.push(l);
}
paras.push(cur);
const bad = new Set();
for (const p of paras.map((x) => x.join('\n')).filter((p) => /stats/i.test(p))) {
for (const m of p.matchAll(/`([^`\n]+)`/g)) {
if (PROFILE_FIELDS.includes(m[1]) && !written.has(m[1])) bad.add(m[1]);
}
}
return [...bad].sort();
}
for (const filename of COMMAND_FILES) {
test(`SC #11(b): commands/${filename} stats prose promises no profile key its stats record does not write`, () => {
const content = readFileSync(join(REPO_ROOT, 'commands', filename), 'utf-8');
const docs = statsHeredocs(content);
assert.equal(docs.length, 1, `${filename}: expected exactly one stats-append.mjs heredoc; found ${docs.length}`);
assert.ok(docs[0].keys.has('ts'), `${filename}: the stats heredoc's keys were not read`);
assert.deepEqual(unwrittenStatsKeys(content, docs[0].keys), [],
`${filename} stats prose promises keys its stats record does not write`);
assert.doesNotMatch(content, /profile_source/,
`${filename} names profile_source, a key no stats record writes`);
});
}
test('SC #11(b): commands/trekplan.md stats prose names neither phase_models nor parallel_agents, which its record does not write', () => {
const content = readFileSync(join(REPO_ROOT, 'commands', 'trekplan.md'), 'utf-8');
const docs = statsHeredocs(content);
assert.equal(docs.length, 1, `trekplan.md: expected exactly one stats-append.mjs heredoc; found ${docs.length}`);
assert.equal(docs[0].keys.has('phase_models'), false, 'the trekplan stats record now writes phase_models: update this test');
assert.equal(docs[0].keys.has('parallel_agents'), false, 'the trekplan stats record now writes parallel_agents: update this test');
const bad = unwrittenStatsKeys(content, docs[0].keys);
assert.ok(!bad.includes('phase_models'), 'trekplan.md stats prose promises phase_models');
assert.ok(!bad.includes('parallel_agents'), 'trekplan.md stats prose promises parallel_agents');
});
test('SC #11(b): the stats-key checker reports a planted paragraph that promises `profile`', () => {
const written = statsHeredocs(readFileSync(join(REPO_ROOT, 'commands', 'trekplan.md'), 'utf-8'))[0].keys;
assert.deepEqual(unwrittenStatsKeys('Stats rows carry `profile`.', written), ['profile']);
});
// --- Step 9: the five STORM measurement fields -----------------------------
// Four numerics + one low-cardinality label. Without `effort` there is no axis
// to group high-vs-standard runs on, and the measurement gate cannot be
// computed at all.
const STORM_FIELDS = [
'effort',
'unique_sources',
'dimensions_baseline',
'conv_turns',
'empty_turns',
];
test('Step 9: a standard-run trekresearch record parses and survives applyFieldAllowlist', async () => {
const { applyFieldAllowlist } = await import('../../lib/exporters/field-allowlist.mjs');
// A standard run: loop never armed, so no discovered dimensions and no turns.
const raw = JSON.parse(JSON.stringify({
_schema_id: 'trekresearch',
ts: '2026-08-09T12:00:00.000Z',
slug: 'add-auth',
mode: 'default',
scope: 'both',
engine: 'swarm',
question: 'free prose that must never reach the exporter',
project_dir: '/Users/somebody/repos/x',
brief_path: '/Users/somebody/repos/x/brief.md',
dimensions: 4,
dimensions_baseline: 4,
conv_turns: 0,
empty_turns: 0,
unique_sources: 11,
effort: 'standard',
agents_local: 5,
agents_external: 4,
gemini_used: false,
confidence: 0.8,
contradictions: 1,
open_questions: 2,
profile: 'premium',
profile_source: 'default',
}));
assert.equal(raw.dimensions_baseline, raw.dimensions,
'a standard run discovers no dimensions — baseline must equal the final count');
const out = applyFieldAllowlist(raw, 'trekresearch');
for (const field of STORM_FIELDS) {
assert.ok(field in out, `${field} must survive the trekresearch allowlist`);
}
assert.equal(out.conv_turns, 0);
assert.equal(out.empty_turns, 0);
assert.equal(out.effort, 'standard');
// Deny-by-omission must still hold for the PII-ish fields.
for (const denied of ['question', 'project_dir', 'brief_path']) {
assert.equal(denied in out, false, `${denied} must NOT reach the exporter`);
}
});
test('Step 9: commands/trekresearch.md prose names all five measurement fields', () => {
const content = readFileSync(join(REPO_ROOT, 'commands', 'trekresearch.md'), 'utf-8');
for (const field of STORM_FIELDS) {
assert.ok(content.includes(field),
`trekresearch.md prose must name ${field} — an emitted-but-undocumented field is unauditable`);
}
});
test('Step 9: tests/fixtures/jsonl-schemas.md trekresearch row lists the five fields', () => {
const doc = readFileSync(join(REPO_ROOT, 'tests', 'fixtures', 'jsonl-schemas.md'), 'utf-8');
const row = doc.split('\n').find(l => l.startsWith('| trekresearch-stats '));
assert.ok(row, 'jsonl-schemas.md is missing the trekresearch-stats row');
for (const field of STORM_FIELDS) {
assert.ok(row.includes(field), `authoring reference must list ${field}`);
}
});
test('SC #11(b): commands/trekresearch.md prose mentions external_research_enabled', () => {
const content = readFileSync(join(REPO_ROOT, 'commands', 'trekresearch.md'), 'utf-8');
assert.match(content, /external_research_enabled/,
'trekresearch.md prose must mention external_research_enabled (additive stats field)');
});

14
tests/lib/removed-surfaces.test.mjs
View file

@ -47,11 +47,21 @@ export const GROUPS = {
// Groups whose removal stage has not run yet. Each stage of R-b empties its
// own entry; REMOVED_SURFACES_ALL=1 checks every group (the red proof).
const PENDING = new Set(['R1', 'R2', 'R3']);
const PENDING = new Set(['R2', 'R3']);
// Named exceptions: "path:line" -> reason. Each must still hit (a stale
// exemption fails below).
const EXEMPT = new Map([]);
const R2_FILE = 'R2 removes this file (the STORM gate reads the old trekresearch stats file); drop the entry with it';
const EXEMPT = new Map([
['docs/storm-measurement.md:80', R2_FILE],
['docs/storm-measurement.md:97', R2_FILE],
['docs/storm-measurement.md:100', R2_FILE],
['docs/storm-measurement.md:109', R2_FILE],
['scripts/storm-measure.mjs:4', R2_FILE],
['scripts/storm-measure.mjs:34', R2_FILE],
['scripts/storm-measure.mjs:47', R2_FILE],
['tests/scripts/storm-measure.test.mjs:8', R2_FILE],
]);
const matches = (term, line) => (typeof term === 'string' ? line.includes(term) : term.test(line));

4
tests/lib/review-630.test.mjs
View file

@ -900,7 +900,7 @@ test('review 630 r8 / criteria-runner.mjs:175: a # inside a word is kept by drop
'the doc says the allowlist still refuses the word');
});
test('review 630 r8 / criteria-runner.mjs:890: no CLI compares process.argv[1] itself, and the three that did work through a symlinked path with a space', () => {
test('review 630 r8 / criteria-runner.mjs:890: no CLI compares process.argv[1] itself, and criteria-runner.mjs works through a symlinked path with a space', () => {
const files = spawnSync('git', ['ls-files', 'lib', 'scripts', 'hooks'], { cwd: ROOT, encoding: 'utf8' }).stdout.split('\n').filter((p) => p.endsWith('.mjs'));
assert.deepEqual(files.filter((p) => p !== 'lib/util/is-main.mjs' && /process\.argv\[1\]/.test(read(p))), [],
'only lib/util/is-main.mjs reads process.argv[1]');
@ -908,7 +908,7 @@ test('review 630 r8 / criteria-runner.mjs:890: no CLI compares process.argv[1] i
const link = join(base, 'voyage link');
try {
symlinkSync(ROOT, link);
for (const cli of ['lib/verification/criteria-runner.mjs', 'scripts/comparison-gate.mjs', 'scripts/end-state-gate.mjs']) {
for (const cli of ['lib/verification/criteria-runner.mjs']) {
const r = spawnSync(process.execPath, [join(link, cli), '--no-such-flag'], { encoding: 'utf8', input: '', timeout: 30_000 });
assert.equal(r.status, 2, `${cli} through "${link}": ${r.stdout}${r.stderr}`);
assert.match(r.stderr, /unknown argument --no-such-flag/);

2
tests/lib/review-prose-pins.test.mjs
View file

@ -255,7 +255,7 @@ test('[FXL] /trekreview Phase 8.8 runs review-answers.mjs pending and record, as
assert.match(f, /`"committed": false`/, 'a failed commit is said in one line');
assert.match(f, /ask that finding again once/, 'a refusal asks that finding again once');
// Phase 8 says the review-unanswered exit 2 is expected before Phase 8.8.
const p8 = section(cmd, '## Phase 8 — Validate output + stats');
const p8 = section(cmd, '## Phase 8 — Validate output');
assert.ok(sentencesWith(p8, '`review-unanswered:`').some((s) => /Phase 8\.8/.test(s) && /expected/.test(s)),
'Phase 8 says an exit 2 with review-unanswered: before Phase 8.8 is expected');
});

8
tests/lib/run-coupling-prose-pins.test.mjs
View file

@ -63,11 +63,11 @@ test('run coupling: the writing-criteria quiet case is owned by run, with no sta
// also one with no app to launch (the "no — {reason}" form).
const EXEC = 'commands/trekexecute.md';
/** Phase 8 of /trekexecute, read up to Phase 9 (its report fence holds its own `## ` heading). */
/** Phase 8 of /trekexecute, read up to Hard rules (its report fence holds its own `## ` heading). */
function phase8(text) {
const start = text.indexOf('## Phase 8 — Final report');
const end = text.indexOf('## Phase 9');
assert.ok(start !== -1 && end > start, 'Phase 8 — Final report and Phase 9 must both exist, in that order');
const end = text.indexOf('## Hard rules');
assert.ok(start !== -1 && end > start, 'Phase 8 — Final report and Hard rules must both exist, in that order');
return text.slice(start, end);
}
@ -151,7 +151,7 @@ test('[E14] Phase 8 reads run_seen from progress.json', () => {
assert.equal(rec.length, 1, `Phase 7 must hold one "Record \`run_seen\`" sentence; found ${rec.length}`);
assert.match(rec[0], /\bin the progress file\b|\bto the progress file\b|`progress\.json`/,
`Phase 7 records run_seen without writing it to the progress file: ${rec[0]}`);
const p8 = between(EXEC, '## Phase 8 — Final report', '## Phase 9 — Stats tracking');
const p8 = between(EXEC, '## Phase 8 — Final report', '## Hard rules');
const hits = sentencesWith(p8, '`run_seen`');
const ok = hits.find((s) => /\bread\b|\breads\b/.test(s)
&& /progress file|`progress\.json`/.test(s)

348
tests/lib/stats-event-emit.test.mjs
View file

@ -1,348 +0,0 @@
// tests/lib/stats-event-emit.test.mjs
// Cover lib/stats/event-emit.mjs:
// - emit appends a JSONL line with required ISO-8601 ts
// - known_event flag distinguishes recognized vs unknown events
// - missing CLAUDE_PLUGIN_DATA falls back to the directory stats-append and
// scripts/yardstick.mjs use, and a failed write never throws
// - CLI shim parses --payload JSON, writes via emit(), exits 1 on any
// written:false, and runs through a symlink or a path with a space
// - concurrent appends don't corrupt the file (smoke test)
//
// Isolation: every test that reaches the fallback pins HOME to a temp dir and
// empties CLAUDE_PLUGIN_DATA. `env: {}` is never used - resolveStatsDir would fall
// back to os.homedir(), the operator's REAL data directory.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { execFileSync, spawnSync } from 'node:child_process';
import { cpSync, mkdtempSync, mkdirSync, rmSync, readFileSync, existsSync, symlinkSync, writeFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { tmpdir } from 'node:os';
import { fileURLToPath } from 'node:url';
import { emit, buildRecord, resolveStatsPath, KNOWN_EVENTS } from '../../lib/stats/event-emit.mjs';
import { resolveStatsDir } from '../../lib/stats/stats-append.mjs';
import { childEnv } from '../helpers/child-env.mjs';
const HERE = dirname(fileURLToPath(import.meta.url));
const SHIM = join(HERE, '..', '..', 'lib', 'stats', 'event-emit.mjs');
const ISO_8601_RE = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
function tmp(prefix = 'stats-event-emit-') {
return mkdtempSync(join(tmpdir(), prefix));
}
const ROOT = join(HERE, '..', '..');
const DATA_REL = join('.claude', 'plugins', 'data', 'voyage-ktg-plugin-marketplace');
/** process.env with HOME pinned and CLAUDE_PLUGIN_DATA empty: never the real data dir. */
function isolatedEnv(home) {
const env = childEnv({ CLAUDE_PLUGIN_DATA: '', HOME: home });
delete env.CLAUDE_COORD_DIR;
return env;
}
function runShim(cli, args, env) {
return spawnSync(process.execPath, [cli, ...args], { encoding: 'utf-8', env });
}
function records(file) {
return readFileSync(file, 'utf-8').trim().split('\n').map((l) => JSON.parse(l));
}
test('KNOWN_EVENTS contains plan-v2 spec set', () => {
for (const e of ['brief-approved', 'main-merge-gate', 'user_input']) {
assert.ok(KNOWN_EVENTS.has(e), `missing recognized event: ${e}`);
}
});
test('buildRecord emits ISO-8601 ts (REQUIRED per SC4)', () => {
const r = buildRecord('brief-approved', { foo: 1 });
assert.match(r.ts, ISO_8601_RE);
assert.equal(r.event, 'brief-approved');
assert.equal(r.known_event, true);
assert.deepEqual(r.payload, { foo: 1 });
});
test('buildRecord marks unrecognized events known_event: false', () => {
const r = buildRecord('totally-made-up-event');
assert.equal(r.known_event, false);
assert.deepEqual(r.payload, {});
});
test('buildRecord rejects empty event name', () => {
assert.throws(() => buildRecord(''), TypeError);
assert.throws(() => buildRecord(null), TypeError);
});
test('emit appends one JSONL line per call', () => {
const dir = tmp();
try {
const path = join(dir, 'stats.jsonl');
const r1 = emit('brief-approved', { ok: true }, { path });
const r2 = emit('main-merge-gate', { branch: 'main' }, { path });
assert.equal(r1.written, true);
assert.equal(r2.written, true);
const lines = readFileSync(path, 'utf-8').trim().split('\n');
assert.equal(lines.length, 2);
const a = JSON.parse(lines[0]);
const b = JSON.parse(lines[1]);
assert.match(a.ts, ISO_8601_RE);
assert.match(b.ts, ISO_8601_RE);
assert.equal(a.event, 'brief-approved');
assert.equal(b.event, 'main-merge-gate');
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test('emit creates the stats directory on demand', () => {
const dir = tmp();
try {
const path = join(dir, 'nested', 'stats.jsonl');
const r = emit('user_input', {}, { path });
assert.equal(r.written, true);
assert.ok(existsSync(path));
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
// Inverted contract (was: "returns { written: false } (silent skip)"). The silent
// skip is how Phase 8's events never landed: the Bash tool env carries no
// CLAUDE_PLUGIN_DATA.
test('emit with no CLAUDE_PLUGIN_DATA lands in the fallback dir under HOME', () => {
const home = tmp();
try {
const r = emit('brief-approved', {}, { env: { HOME: home } });
assert.equal(r.written, true, JSON.stringify(r));
assert.equal(r.path, join(home, DATA_REL, 'trekexecute-stats.jsonl'));
assert.equal(records(r.path).length, 1);
} finally {
rmSync(home, { recursive: true, force: true });
}
});
// Inverted contract (was: env {} as the "unwritable" case - after the fallback an
// empty env resolves to the real home, so it is no longer unwritable).
test('emit never throws when stats path is unwritable', () => {
const dir = tmp();
try {
const file = join(dir, 'a-file');
writeFileSync(file, 'not a directory');
let r;
assert.doesNotThrow(() => { r = emit('user_input', { foo: 'bar' }, { path: join(file, 'x.jsonl') }); });
assert.equal(r.written, false);
assert.ok(r.reason && r.reason.length > 0, JSON.stringify(r));
mkdirSync(join(dir, '.claude', 'plugins', 'data'), { recursive: true });
writeFileSync(join(dir, DATA_REL), 'not a directory');
assert.doesNotThrow(() => { r = emit('user_input', {}, { env: { HOME: dir } }); });
assert.equal(r.written, false);
assert.ok(r.reason && r.reason.length > 0, JSON.stringify(r));
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
// Inverted contract (was: resolveStatsPath({}) === null).
test('resolveStatsPath honors CLAUDE_PLUGIN_DATA, else the stats-append directory', () => {
const r = resolveStatsPath({ CLAUDE_PLUGIN_DATA: '/var/data/plugin' });
assert.equal(r, '/var/data/plugin/trekexecute-stats.jsonl');
assert.equal(resolveStatsPath({ HOME: '/h' }), join(resolveStatsDir({ HOME: '/h' }), 'trekexecute-stats.jsonl'));
assert.equal(resolveStatsPath({ HOME: '/h', CLAUDE_PLUGIN_DATA: '' }), join('/h', DATA_REL, 'trekexecute-stats.jsonl'));
});
// gates_mode_flag_override stays a known event; no command emits it since fix round 4.5a
test('the Phase 8 and override events are known events', () => {
for (const e of ['main-merge-approved', 'main-merge-declined', 'gates_mode_flag_override']) {
assert.equal(buildRecord(e).known_event, true, `${e} is a known event`);
}
});
// Regression guard: one resolver - event-emit carries no second copy of the fallback path.
test('event-emit has no copy of the fallback directory name', () => {
assert.doesNotMatch(readFileSync(SHIM, 'utf-8'), /voyage-ktg-plugin-marketplace/);
});
test('CLI shim writes via emit when CLAUDE_PLUGIN_DATA is set', () => {
const dir = tmp();
try {
execFileSync(process.execPath, [
SHIM, '--event', 'brief-approved', '--payload', '{"foo":42}',
], {
env: childEnv({ CLAUDE_PLUGIN_DATA: dir }),
encoding: 'utf-8',
});
const path = join(dir, 'trekexecute-stats.jsonl');
assert.ok(existsSync(path));
const line = readFileSync(path, 'utf-8').trim();
const parsed = JSON.parse(line);
assert.equal(parsed.event, 'brief-approved');
assert.deepEqual(parsed.payload, { foo: 42 });
assert.match(parsed.ts, ISO_8601_RE);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
// Inverted contract (was: "... payload-not-json (exit 0)"): every written:false exits 1.
test('CLI shim with malformed --payload returns reason payload-not-json (exit 1)', () => {
const home = tmp();
try {
const r = runShim(SHIM, ['--event', 'user_input', '--payload', 'not-json{{'], isolatedEnv(home));
assert.equal(r.status, 1, r.stdout + r.stderr);
const parsed = JSON.parse(r.stdout.trim());
assert.equal(parsed.written, false);
assert.equal(parsed.reason, 'payload-not-json');
} finally {
rmSync(home, { recursive: true, force: true });
}
});
// `--payload -` reads the payload from stdin, so the command prose can pass it
// through a quoted heredoc: the shell expands nothing, and a plan path with a
// quote character can no longer close a single-quoted argument.
test('CLI shim with --payload - reads the payload from stdin, quote characters and all', () => {
const home = tmp();
try {
const plan = ".claude/projects/it's a'$(touch PWNED)'b/plan.md";
const r = spawnSync(process.execPath, [SHIM, '--event', 'main-merge-gate', '--payload', '-'],
{ encoding: 'utf-8', env: isolatedEnv(home), input: JSON.stringify({ plan, wave_count: 1 }) });
assert.equal(r.status, 0, r.stdout + r.stderr);
const [rec] = records(join(home, DATA_REL, 'trekexecute-stats.jsonl'));
assert.deepEqual(rec.payload, { plan, wave_count: 1 });
} finally {
rmSync(home, { recursive: true, force: true });
}
});
test('CLI shim with --payload - and a stdin that is not JSON returns payload-not-json (exit 1)', () => {
const home = tmp();
try {
const r = spawnSync(process.execPath, [SHIM, '--event', 'main-merge-gate', '--payload', '-'],
{ encoding: 'utf-8', env: isolatedEnv(home), input: 'not-json{{' });
assert.equal(r.status, 1, r.stdout + r.stderr);
assert.equal(JSON.parse(r.stdout.trim()).reason, 'payload-not-json');
assert.ok(!existsSync(join(home, DATA_REL, 'trekexecute-stats.jsonl')));
} finally {
rmSync(home, { recursive: true, force: true });
}
});
// The emit() docstring said failures are "swallowed silently" after the contract
// became {written:false, reason}.
test('emit() is not documented as swallowing failures silently', () => {
assert.doesNotMatch(readFileSync(SHIM, 'utf-8'), /swallowed silently/);
});
test('CLI: with CLAUDE_PLUGIN_DATA empty the event lands under HOME', () => {
const home = tmp();
try {
const r = runShim(SHIM, ['--event', 'main-merge-gate', '--payload', '{}'], isolatedEnv(home));
assert.equal(r.status, 0, r.stdout + r.stderr);
assert.equal(JSON.parse(r.stdout.trim()).written, true, r.stdout);
const recs = records(join(home, DATA_REL, 'trekexecute-stats.jsonl'));
assert.equal(recs.length, 1);
assert.equal(recs[0].event, 'main-merge-gate');
assert.equal(recs[0].known_event, true);
assert.deepEqual(recs[0].payload, {});
} finally {
rmSync(home, { recursive: true, force: true });
}
});
test('CLI: with CLAUDE_PLUGIN_DATA absent the event lands under HOME', () => {
const home = tmp();
try {
const env = isolatedEnv(home);
delete env.CLAUDE_PLUGIN_DATA;
const r = runShim(SHIM, ['--event', 'main-merge-gate', '--payload', '{}'], env);
assert.equal(r.status, 0, r.stdout + r.stderr);
assert.equal(records(join(home, DATA_REL, 'trekexecute-stats.jsonl')).length, 1);
} finally {
rmSync(home, { recursive: true, force: true });
}
});
test('CLI: an unwritable target exits 1 with a reason', () => {
const home = tmp();
try {
mkdirSync(join(home, '.claude', 'plugins', 'data'), { recursive: true });
writeFileSync(join(home, DATA_REL), 'not a directory');
const r = runShim(SHIM, ['--event', 'main-merge-gate', '--payload', '{}'], isolatedEnv(home));
assert.equal(r.status, 1, r.stdout + r.stderr);
const out = JSON.parse(r.stdout.trim());
assert.equal(out.written, false);
assert.ok(out.reason && out.reason.length > 0, r.stdout);
} finally {
rmSync(home, { recursive: true, force: true });
}
});
test('CLI: a usage error exits 1 with a reason', () => {
const home = tmp();
try {
const r = runShim(SHIM, [], isolatedEnv(home));
assert.equal(r.status, 1, r.stdout + r.stderr);
const out = JSON.parse(r.stdout.trim());
assert.equal(out.written, false);
assert.match(out.reason, /usage/);
} finally {
rmSync(home, { recursive: true, force: true });
}
});
test('CLI: event-emit writes when it is run through a symlink', () => {
const home = tmp();
const links = tmp('stats-event-emit-link-');
try {
const link = join(links, 'event-emit.mjs');
symlinkSync(SHIM, link);
const r = runShim(link, ['--event', 'main-merge-gate', '--payload', '{}'], isolatedEnv(home));
assert.ok(r.stdout.trim().length > 0, `no result line - the CLI guard did not fire; stderr: ${r.stderr}`);
assert.equal(r.status, 0, r.stdout + r.stderr);
assert.ok(existsSync(join(home, DATA_REL, 'trekexecute-stats.jsonl')), `no record landed; stdout: "${r.stdout}"`);
} finally {
rmSync(home, { recursive: true, force: true });
rmSync(links, { recursive: true, force: true });
}
});
test('CLI: event-emit writes when its path contains a space', () => {
const home = tmp();
const spaced = tmp('stats event emit space-');
try {
cpSync(join(ROOT, 'lib'), join(spaced, 'with space', 'lib'), { recursive: true });
const cli = join(spaced, 'with space', 'lib', 'stats', 'event-emit.mjs');
const r = runShim(cli, ['--event', 'main-merge-gate', '--payload', '{}'], isolatedEnv(home));
assert.ok(r.stdout.trim().length > 0, `no result line - the CLI guard did not fire; stderr: ${r.stderr}`);
assert.equal(r.status, 0, r.stdout + r.stderr);
assert.ok(existsSync(join(home, DATA_REL, 'trekexecute-stats.jsonl')), `no record landed; stdout: "${r.stdout}"`);
} finally {
rmSync(home, { recursive: true, force: true });
rmSync(spaced, { recursive: true, force: true });
}
});
test('concurrent appends do not corrupt JSONL (smoke)', async () => {
const dir = tmp();
try {
const path = join(dir, 'stats.jsonl');
const N = 25;
await Promise.all(
Array.from({ length: N }, (_, i) =>
Promise.resolve().then(() => emit('user_input', { i }, { path })),
),
);
const lines = readFileSync(path, 'utf-8').trim().split('\n');
assert.equal(lines.length, N);
for (const l of lines) {
const parsed = JSON.parse(l); // throws if any line is corrupt
assert.ok('ts' in parsed);
assert.equal(parsed.event, 'user_input');
}
} finally {
rmSync(dir, { recursive: true, force: true });
}
});

453
tests/lib/stats-landing.test.mjs
View file

@ -1,453 +0,0 @@
// tests/lib/stats-landing.test.mjs
// Veikart steg 1, økt 2's countable form: after a real /trekbrief → /trekplan run
// there must be (1) a `brief-approved` record with the slug, (2) a trekbrief-stats
// record with `intent_approved: true`, and (3) a trekplan-stats record with the
// same slug — all in the directory scripts/yardstick.mjs reads.
//
// The Bash tool's env carries no CLAUDE_PLUGIN_DATA. Phase 7 of /trekbrief and
// Phase 12 of /trekplan used to say "if ${CLAUDE_PLUGIN_DATA} is not set, skip
// silently" — the same silent skip that left brief-approved at 0 records. These
// tests execute the command lines the prose spells, with CLAUDE_PLUGIN_DATA
// EMPTY, and require the records to land in the yardstick's directory.
//
// The trekbrief record also carries the coord-queue order the run is working on
// (order_id, order_claimed_at, order_reason). The WRITER adds those from the
// queue, so these tests run the line inside a temp git repo against a temp coord
// dir - and every spawn drops an inherited CLAUDE_COORD_DIR, because the real
// queue can hold a claimed order while the suite runs.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { spawnSync } from 'node:child_process';
import { cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { childEnv } from '../helpers/child-env.mjs';
const HERE = dirname(fileURLToPath(import.meta.url));
const ROOT = join(HERE, '..', '..');
const read = (rel) => readFileSync(join(ROOT, rel), 'utf8');
const DATA_REL = join('.claude', 'plugins', 'data', 'voyage-ktg-plugin-marketplace');
/** The text of `## <heading>` up to the next `## ` heading. */
function section(rel, heading) {
const text = read(rel);
const start = text.indexOf(`\n## ${heading}`);
assert.ok(start >= 0, `${rel} must have "## ${heading}"`);
const next = text.indexOf('\n## ', start + 1);
return next < 0 ? text.slice(start) : text.slice(start, next);
}
/** The one line in `sec` that runs stats-append.mjs for `kind`. */
function appendLine(sec, kind, rel) {
const lines = sec.split('\n').filter((l) =>
/^\s*node \$\{CLAUDE_PLUGIN_ROOT\}\/lib\/stats\/stats-append\.mjs\b/.test(l) && l.includes(` ${kind}`));
assert.equal(lines.length, 1, `${rel} must carry exactly one \`stats-append.mjs ${kind}\` line; found ${lines.length}`);
return lines[0].trim().replace(/\s*<<-?'?JSON'?\s*$/, '');
}
/** process.env without CLAUDE_COORD_DIR: a test never reads the real coord queue. */
function baseEnv() {
const env = childEnv();
delete env.CLAUDE_COORD_DIR;
return env;
}
/** Run a prose command line with a record on stdin, CLAUDE_PLUGIN_DATA empty, HOME = home. */
function runLine(line, record, home, { cwd, env: extra = {} } = {}) {
const cmd = line.split('${CLAUDE_PLUGIN_ROOT}').join(ROOT);
return spawnSync('bash', ['-c', cmd], {
encoding: 'utf8',
cwd,
input: JSON.stringify(record),
env: { ...baseEnv(), CLAUDE_PLUGIN_DATA: '', HOME: home, ...extra },
});
}
/** A git repo whose directory is named `name` (its mailbox), under a fresh temp parent. */
function gitRepo(name) {
const parent = mkdtempSync(join(tmpdir(), 'stats-landing-git-'));
const dir = join(parent, name);
mkdirSync(dir);
assert.equal(spawnSync('git', ['init', '-q'], { cwd: dir }).status, 0);
return { parent, dir };
}
/** A coord dir holding claimed orders: `{ mailbox: [id, ...] }`, each with a .claim sidecar. */
function coordWith(claims) {
const coord = mkdtempSync(join(tmpdir(), 'stats-landing-coord-'));
for (const [mailbox, ids] of Object.entries(claims)) {
const dir = join(coord, mailbox, 'orders', 'claimed');
mkdirSync(dir, { recursive: true });
for (const id of ids) {
writeFileSync(join(dir, `${id}.md`), `---\nfrom: .claude\nto: ${mailbox}\norder-id: ${id}\ndate: 2026-10-04T10:00:00Z\n---\n\n# ORDER ${id}\n`);
writeFileSync(join(dir, `${id}.claim`), 'claimed-at: 2026-10-04T17:10:51Z\nclaimed-by-pid: 1\n');
}
}
return coord;
}
function records(file) {
return readFileSync(file, 'utf8').trim().split('\n').filter(Boolean).map((l) => JSON.parse(l));
}
const TB = 'commands/trekbrief.md';
const TP = 'commands/trekplan.md';
test('M6: /trekbrief Phase 7 writes through stats-append (code), not "skip silently" (prose)', () => {
const sec = section(TB, 'Phase 7');
appendLine(sec, 'trekbrief', TB);
assert.doesNotMatch(sec, /skip silently/i, 'a silent skip is how the record never landed');
assert.match(sec, /"intent_approved"/);
assert.match(sec, /"slug"/);
});
test('M6: /trekplan Phase 12 writes through stats-append (code), not "skip silently" (prose)', () => {
const sec = section(TP, 'Phase 12');
appendLine(sec, 'trekplan', TP);
assert.doesNotMatch(sec, /skip (tracking )?silently/i);
assert.match(sec, /"slug"/);
});
test('M6: with CLAUDE_PLUGIN_DATA empty the trekbrief record (intent_approved) lands where the yardstick reads', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
try {
const line = appendLine(section(TB, 'Phase 7'), 'trekbrief', TB);
const r = runLine(line, { ts: '2026-09-23T10:00:00Z', slug: 'dark-mode-toggle', intent_approved: true }, home);
assert.equal(r.status, 0, `stats-append failed: ${r.stdout} ${r.stderr}`);
const file = join(home, DATA_REL, 'trekbrief-stats.jsonl');
assert.ok(existsSync(file), `record must land in ${file}`);
const recs = records(file);
assert.equal(recs.length, 1);
assert.equal(recs[0].intent_approved, true);
assert.equal(recs[0].slug, 'dark-mode-toggle');
} finally { rmSync(home, { recursive: true, force: true }); }
});
test('M6: a failed write is reported, never silent (exit 1 + reason on stdout)', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
try {
// the data dir path is occupied by a FILE, so the directory cannot be created
mkdirSync(join(home, '.claude', 'plugins', 'data'), { recursive: true });
writeFileSync(join(home, DATA_REL), 'not a directory');
const line = appendLine(section(TB, 'Phase 7'), 'trekbrief', TB);
const r = runLine(line, { ts: '2026-09-23T10:00:00Z', slug: 'x', intent_approved: false }, home);
assert.equal(r.status, 1);
const out = JSON.parse(r.stdout);
assert.equal(out.written, false);
assert.ok(out.reason && out.reason.length > 0, r.stdout);
} finally { rmSync(home, { recursive: true, force: true }); }
});
test('M6: stats-append refuses a record that is not a JSON object, and an unknown kind', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
try {
const cli = join(ROOT, 'lib', 'stats', 'stats-append.mjs');
const env = { ...baseEnv(), CLAUDE_PLUGIN_DATA: '', HOME: home };
const bad = spawnSync('node', [cli, 'trekbrief'], { encoding: 'utf8', input: 'not json', env });
assert.equal(bad.status, 1, bad.stdout + bad.stderr);
assert.equal(JSON.parse(bad.stdout).written, false);
const unknown = spawnSync('node', [cli, 'no-such-command'], { encoding: 'utf8', input: '{}', env });
assert.equal(unknown.status, 1);
assert.ok(!existsSync(join(home, DATA_REL, 'no-such-command-stats.jsonl')));
} finally { rmSync(home, { recursive: true, force: true }); }
});
test('M6: økt 2\'s countable form lands end to end with CLAUDE_PLUGIN_DATA empty (stamp → trekbrief → trekplan)', async () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
const proj = mkdtempSync(join(tmpdir(), 'stats-landing-proj-'));
try {
writeFileSync(join(proj, 'brief.md'), `---
type: trekbrief
brief_version: "2.1"
task: "Add a dark-mode toggle to a recipe app"
slug: dark-mode-toggle
---
## Intent
Readers cook in dim kitchens and the bright page glares.
## Goal
A toggle in settings, defaulting to the system theme.
`);
// (1) the Phase 4h stamp line, as trekbrief.md spells it
const stampLine = read(TB).split('\n').find((l) =>
/^\s*node \$\{CLAUDE_PLUGIN_ROOT\}\/lib\/validators\/intent-approval\.mjs\b.*--stamp/.test(l));
assert.ok(stampLine, 'trekbrief.md must carry the --stamp line');
let cmd = stampLine.trim().split('${CLAUDE_PLUGIN_ROOT}').join(ROOT);
cmd = cmd.split('{BRIEF_PATH}').join(join(proj, 'brief.md')).split('{PROJECT_DIR}').join(proj);
const s = spawnSync('bash', ['-c', cmd], { encoding: 'utf8', env: { ...baseEnv(), CLAUDE_PLUGIN_DATA: '', HOME: home } });
assert.equal(s.status, 0, s.stdout + s.stderr);
// (2) and (3): the stats lines
assert.equal(runLine(appendLine(section(TB, 'Phase 7'), 'trekbrief', TB),
{ ts: '2026-09-23T10:00:00Z', slug: 'dark-mode-toggle', intent_approved: true }, home).status, 0);
assert.equal(runLine(appendLine(section(TP, 'Phase 12'), 'trekplan', TP),
{ ts: '2026-09-23T10:05:00Z', slug: 'dark-mode-toggle', outcome: 'save' }, home).status, 0);
const { loadStats } = await import(join(ROOT, 'scripts', 'yardstick.mjs'));
const stats = loadStats(join(home, DATA_REL));
const all = stats.files.flatMap((f) => f.records.map((r) => ({ file: f.name, r })));
const approved = all.filter(({ r }) => r.event === 'brief-approved' && r.payload?.slug === 'dark-mode-toggle');
const brief = all.filter(({ file, r }) => file === 'trekbrief-stats.jsonl' && r.intent_approved === true && r.slug === 'dark-mode-toggle');
const plan = all.filter(({ file, r }) => file === 'trekplan-stats.jsonl' && r.slug === 'dark-mode-toggle');
assert.equal(approved.length, 1, 'brief-approved with slug');
assert.equal(brief.length, 1, 'trekbrief record with intent_approved: true');
assert.equal(plan.length, 1, 'trekplan record with the same slug');
} finally {
rmSync(home, { recursive: true, force: true });
rmSync(proj, { recursive: true, force: true });
}
});
test('M6: stats-append and intent-approval resolve the same data dir as the yardstick', async () => {
const { resolveStatsDir } = await import(join(ROOT, 'lib', 'stats', 'stats-append.mjs'));
const { resolveApprovalDataDir } = await import(join(ROOT, 'lib', 'validators', 'intent-approval.mjs'));
const env = { HOME: '/home/someone', CLAUDE_PLUGIN_DATA: '' };
assert.equal(resolveStatsDir(env), join('/home/someone', DATA_REL));
assert.equal(resolveApprovalDataDir(env), resolveStatsDir(env));
assert.equal(resolveStatsDir({ HOME: '/h', CLAUDE_PLUGIN_DATA: '/x/y' }), '/x/y');
});
// --- the order the run is working on (written by the writer, never the model) -------
const ORDER_KEYS = ['order_id', 'order_claimed_at', 'order_reason'];
/** Run Phase 7's line in `repo` against `coord` (or none) and return the one landed record. */
function landTrekbrief(record, { repo, coord, home }) {
const line = appendLine(section(TB, 'Phase 7'), 'trekbrief', TB);
const env = coord ? { CLAUDE_COORD_DIR: coord } : {};
const r = runLine(line, record, home, { cwd: repo, env });
assert.equal(r.status, 0, `stats-append failed: ${r.stdout} ${r.stderr}`);
const recs = records(join(home, DATA_REL, 'trekbrief-stats.jsonl'));
assert.equal(recs.length, 1);
return recs[0];
}
test('order: one claimed order in the repo\'s mailbox lands in the trekbrief record, with its claim time', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
const repo = gitRepo('mbx');
const coord = coordWith({ mbx: ['o-1'] });
try {
const rec = landTrekbrief({ ts: '2026-10-04T18:00:00Z', slug: 'dark-mode-toggle', intent_approved: true },
{ repo: repo.dir, coord, home });
assert.equal(rec.order_id, 'o-1');
assert.equal(rec.order_claimed_at, '2026-10-04T17:10:51Z');
assert.equal(rec.order_reason, null);
assert.equal(rec.slug, 'dark-mode-toggle', 'the record keeps its own slug');
} finally { rmSync(home, { recursive: true, force: true }); rmSync(repo.parent, { recursive: true, force: true }); rmSync(coord, { recursive: true, force: true }); }
});
test('order: the record\'s own project_dir finds the mailbox when the shell stands outside the repo', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
const repo = gitRepo('mbx');
const coord = coordWith({ mbx: ['o-1'] });
try {
mkdirSync(join(repo.dir, '.claude', 'projects', 'x'), { recursive: true });
const rec = landTrekbrief({ ts: '2026-10-04T18:00:00Z', slug: 's', project_dir: 'mbx/.claude/projects/x' },
{ repo: repo.parent, coord, home });
assert.equal(rec.order_id, 'o-1');
assert.equal(rec.order_reason, null);
} finally { rmSync(home, { recursive: true, force: true }); rmSync(repo.parent, { recursive: true, force: true }); rmSync(coord, { recursive: true, force: true }); }
});
test('order: zero claimed orders - the three keys are PRESENT, order_id null, reason none-claimed', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
const repo = gitRepo('mbx');
const coord = coordWith({ other: ['o-9'] });
try {
const rec = landTrekbrief({ ts: '2026-10-04T18:00:00Z', slug: 's' }, { repo: repo.dir, coord, home });
for (const k of ORDER_KEYS) assert.ok(k in rec, `${k} must be present (absent means: written before the recorder existed)`);
assert.equal(rec.order_id, null);
assert.equal(rec.order_reason, 'none-claimed');
} finally { rmSync(home, { recursive: true, force: true }); rmSync(repo.parent, { recursive: true, force: true }); rmSync(coord, { recursive: true, force: true }); }
});
test('order: two claimed orders - order_id null, reason several-claimed:2, never a guess', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
const repo = gitRepo('mbx');
const coord = coordWith({ mbx: ['o-1', 'o-2'] });
try {
const rec = landTrekbrief({ ts: '2026-10-04T18:00:00Z', slug: 's' }, { repo: repo.dir, coord, home });
assert.equal(rec.order_id, null);
assert.equal(rec.order_reason, 'several-claimed:2');
} finally { rmSync(home, { recursive: true, force: true }); rmSync(repo.parent, { recursive: true, force: true }); rmSync(coord, { recursive: true, force: true }); }
});
test('order: order_* keys the model put in the record are replaced by the writer\'s', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
const repo = gitRepo('mbx');
const coord = coordWith({ mbx: ['o-1'] });
try {
const rec = landTrekbrief({ ts: '2026-10-04T18:00:00Z', slug: 's', order_id: 'forged', order_extra: 1 },
{ repo: repo.dir, coord, home });
assert.equal(rec.order_id, 'o-1');
assert.ok(!('order_extra' in rec), 'only the writer\'s three order keys survive');
} finally { rmSync(home, { recursive: true, force: true }); rmSync(repo.parent, { recursive: true, force: true }); rmSync(coord, { recursive: true, force: true }); }
});
test('order: a trekplan record gains no order keys - only /trekbrief records the order', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
const repo = gitRepo('mbx');
const coord = coordWith({ mbx: ['o-1'] });
try {
const r = runLine(appendLine(section(TP, 'Phase 12'), 'trekplan', TP), { ts: '2026-10-04T18:00:00Z', slug: 's' }, home,
{ cwd: repo.dir, env: { CLAUDE_COORD_DIR: coord } });
assert.equal(r.status, 0, r.stdout + r.stderr);
const [rec] = records(join(home, DATA_REL, 'trekplan-stats.jsonl'));
for (const k of ORDER_KEYS) assert.ok(!(k in rec), `${k} must not be on a trekplan record`);
} finally { rmSync(home, { recursive: true, force: true }); rmSync(repo.parent, { recursive: true, force: true }); rmSync(coord, { recursive: true, force: true }); }
});
test('order: with CLAUDE_COORD_DIR unset and HOME a temp dir the real queue is never read - order_id null', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
const repo = gitRepo('voyage');
try {
const rec = landTrekbrief({ ts: '2026-10-04T18:00:00Z', slug: 's' }, { repo: repo.dir, home });
assert.ok('order_id' in rec);
assert.equal(rec.order_id, null);
assert.equal(rec.order_reason, 'coord-missing');
} finally { rmSync(home, { recursive: true, force: true }); rmSync(repo.parent, { recursive: true, force: true }); }
});
test('CLI: stats-append writes when it is run through a symlink', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
const links = mkdtempSync(join(tmpdir(), 'stats-landing-link-'));
try {
const link = join(links, 'stats-append.mjs');
symlinkSync(join(ROOT, 'lib', 'stats', 'stats-append.mjs'), link);
const r = spawnSync('node', [link, 'trekplan'], { encoding: 'utf8', input: '{"slug":"s"}',
env: { ...baseEnv(), CLAUDE_PLUGIN_DATA: '', HOME: home } });
assert.equal(r.status, 0, r.stdout + r.stderr);
assert.ok(existsSync(join(home, DATA_REL, 'trekplan-stats.jsonl')), `no record landed; stdout: "${r.stdout}"`);
} finally { rmSync(home, { recursive: true, force: true }); rmSync(links, { recursive: true, force: true }); }
});
test('CLI: stats-append writes when its path contains a space', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
const spaced = mkdtempSync(join(tmpdir(), 'stats landing space-'));
try {
cpSync(join(ROOT, 'lib'), join(spaced, 'with space', 'lib'), { recursive: true });
const cli = join(spaced, 'with space', 'lib', 'stats', 'stats-append.mjs');
const r = spawnSync('node', [cli, 'trekplan'], { encoding: 'utf8', input: '{"slug":"s"}',
env: { ...baseEnv(), CLAUDE_PLUGIN_DATA: '', HOME: home } });
assert.equal(r.status, 0, r.stdout + r.stderr);
assert.ok(existsSync(join(home, DATA_REL, 'trekplan-stats.jsonl')), `no record landed; stdout: "${r.stdout}"`);
} finally { rmSync(home, { recursive: true, force: true }); rmSync(spaced, { recursive: true, force: true }); }
});
// ---- /trekexecute: every telemetry write lands through a writer ----------------
// Phase 8's three event-emit calls and Phase 9's stats record; the
// gates_mode_flag_override record is gone, since no effort sets gates_mode (#33).
// Phase 9 said "skip silently"; Phase 8's payloads spliced ${PLAN_PATH} (set in an earlier
// Bash call that does not survive) and ${WAVE_COUNT} (never assigned) into JSON,
// so a literal run printed payload-not-json. Every block runs here in a fresh
// shell with NOTHING exported, CLAUDE_PLUGIN_DATA empty and HOME pinned.
const TE = 'commands/trekexecute.md';
/** The fenced bash blocks in `sec` that contain `needle`. */
function fencedBlocksWith(sec, needle) {
const blocks = [];
const re = /```bash\n([\s\S]*?)```/g;
let m;
while ((m = re.exec(sec)) !== null) if (m[1].includes(needle)) blocks.push(m[1]);
return blocks;
}
/** Fill the prose placeholders an executor fills in, then run the block. */
function runBlock(block, home, { planPath = '.claude/projects/2026-10-05-x/plan.md', cwd } = {}) {
const cmd = block.split('${CLAUDE_PLUGIN_ROOT}').join(ROOT)
.split('{plan-path}').join(planPath)
.split('{wave-count}').join('1');
return spawnSync('bash', ['-c', cmd], {
encoding: 'utf8',
cwd,
env: { ...baseEnv(), CLAUDE_PLUGIN_DATA: '', HOME: home },
});
}
test('M6: /trekexecute Phase 9 writes through stats-append (code), never "skip silently" (prose)', () => {
appendLine(section(TE, 'Phase 9'), 'trekexecute', TE);
assert.doesNotMatch(read(TE), /skip (tracking )?silently/i, 'a silent skip is how the record never landed');
});
test('M6: with CLAUDE_PLUGIN_DATA empty the trekexecute record lands where the yardstick reads', () => {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
try {
const line = appendLine(section(TE, 'Phase 9'), 'trekexecute', TE);
const r = runLine(line, { ts: '2026-10-05T10:00:00Z', plan: 'plan.md', result: 'completed' }, home);
assert.equal(r.status, 0, r.stdout + r.stderr);
assert.equal(records(join(home, DATA_REL, 'trekexecute-stats.jsonl')).length, 1);
} finally { rmSync(home, { recursive: true, force: true }); }
});
test('trekexecute.md names no bare ${CLAUDE_PLUGIN_DATA}/...jsonl write target', () => {
const hits = read(TE).split('\n')
.map((l, i) => [i + 1, l]).filter(([, l]) => /\$\{CLAUDE_PLUGIN_DATA\}\/[\w-]+\.jsonl/.test(l));
assert.deepEqual(hits, [], 'every stats or event write goes through a writer that resolves the directory');
});
test('no --payload in trekexecute.md expands a shell variable', () => {
const hits = read(TE).split('\n')
.map((l, i) => [i + 1, l]).filter(([, l]) => /--payload\b.*\$\{/.test(l));
assert.deepEqual(hits, [], 'a payload built from shell state that does not survive between Bash calls never parses');
});
test('Phase 8: each event block, run as written, lands one known event', () => {
const blocks = fencedBlocksWith(section(TE, 'Phase 8'), 'event-emit.mjs');
assert.equal(blocks.length, 3, 'main-merge-gate, main-merge-approved, main-merge-declined');
for (const block of blocks) {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
try {
const r = runBlock(block, home);
assert.equal(r.status, 0, `${block}\n${r.stdout}${r.stderr}`);
assert.equal(JSON.parse(r.stdout.trim()).written, true, r.stdout);
const [rec] = records(join(home, DATA_REL, 'trekexecute-stats.jsonl'));
assert.equal(rec.known_event, true, rec.event);
} finally { rmSync(home, { recursive: true, force: true }); }
}
});
// The plan path is operator-chosen (positional [plan.md], --project <dir>). Spliced
// into a single-quoted argument, a lone quote broke the shell parse (event lost,
// no written:false to report) and a quoted pair let a command substitution run.
test('Phase 8: a plan path with a quote character lands literally and the shell runs nothing in it', () => {
const blocks = fencedBlocksWith(section(TE, 'Phase 8'), 'event-emit.mjs');
assert.equal(blocks.length, 3, 'main-merge-gate, main-merge-approved, main-merge-declined');
for (const plan of [".claude/projects/2026-10-05-it's/plan.md", ".claude/projects/a'$(touch PWNED)'b/plan.md"]) {
for (const block of blocks) {
const home = mkdtempSync(join(tmpdir(), 'stats-landing-'));
try {
const r = runBlock(block, home, { planPath: plan, cwd: home });
assert.ok(!existsSync(join(home, 'PWNED')), `the shell ran a command from the plan path:\n${block}`);
assert.equal(r.status, 0, `${block}\n${r.stdout}${r.stderr}`);
const [rec] = records(join(home, DATA_REL, 'trekexecute-stats.jsonl'));
assert.equal(rec.payload.plan, plan);
} finally { rmSync(home, { recursive: true, force: true }); }
}
}
});
test('gates_mode_flag_override: the Composition rule holds no override record to report on', () => {
const flat = section(TE, 'Composition rule (v5.1)').replace(/\s+/g, ' ');
assert.doesNotMatch(flat, /--event gates_mode_flag_override/, 'no effort sets gates_mode, so there is no override to log');
assert.doesNotMatch(flat, /override record never blocks the run/);
});
test('Phase 8: a written:false is printed in one line and the run continues', () => {
const sec = section(TE, 'Phase 8');
const flat = sec.replace(/\s+/g, ' ');
assert.match(flat, /On `"written": false` \(exit 1\), print its `reason` in one line and continue/);
assert.match(flat, /never chained with `&&`/);
});
test('gates_mode_flag_override: /trekexecute holds no --event gates_mode_flag_override block', () => {
const sec = section(TE, 'Composition rule (v5.1)');
assert.deepEqual(fencedBlocksWith(sec, '--event gates_mode_flag_override'), [],
'high effort sets no gates_mode, so no override is written');
assert.deepEqual(fencedBlocksWith(read(TE), 'gates_mode_flag_override'), []);
assert.doesNotMatch(sec, /\{ts, event, known_event, payload:\{flag, signal_effort\}\}/);
});

188
tests/lib/token-usage.test.mjs
View file

@ -1,188 +0,0 @@
// tests/lib/token-usage.test.mjs
// SKAL-2 Step 1 — pure token-usage parser + cache-aware cost derivation.
//
// Hermetic: no LLM, network, time, or randomness. Fixture transcript exercises
// the dedup-by-requestId (streaming-placeholder) and isSidechain-exclusion
// paths that were verified against a real local transcript at execute time.
// Idiom: tests/lib/coordinator-contract.test.mjs (pure-module node:test style).
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { readFileSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import {
parseTranscriptUsage,
deriveCost,
buildRecord,
upsertSessionRecord,
lastMainChainModel,
PRICE_TABLE,
PRICE_TABLE_VERSION,
} from '../../lib/stats/token-usage.mjs';
const HERE = dirname(fileURLToPath(import.meta.url));
const FIXTURE = join(HERE, '..', 'fixtures', 'token-usage', 'sample-transcript.jsonl');
// ---- parseTranscriptUsage --------------------------------------------------
test('parseTranscriptUsage — sums main-chain final-per-requestId; excludes sidechain + placeholder dup', () => {
const text = readFileSync(FIXTURE, 'utf-8');
const totals = parseTranscriptUsage(text);
// req_A: placeholder {1,1,0,0} then real {1000,200,400,5000} → LAST wins.
// req_B: appears twice identically → dedup → counted once {2000,300,600,1000}.
// req_SIDE (isSidechain:true, all 9999) → EXCLUDED. user/system/malformed → skipped.
assert.deepEqual(totals, {
tokens_input: 3000, // 1000 + 2000 (NOT 1 — placeholder dropped)
tokens_output: 500, // 200 + 300
tokens_cache_creation: 1000, // 400 + 600
tokens_cache_read: 6000, // 5000 + 1000
});
// Sidechain 9999s must not leak into any field.
assert.ok(totals.tokens_input < 9999, 'sidechain tokens leaked into total');
});
test('parseTranscriptUsage — empty / nullish input → zeroed totals', () => {
const zero = { tokens_input: 0, tokens_output: 0, tokens_cache_creation: 0, tokens_cache_read: 0 };
assert.deepEqual(parseTranscriptUsage(''), zero);
assert.deepEqual(parseTranscriptUsage(null), zero);
assert.deepEqual(parseTranscriptUsage(undefined), zero);
});
test('parseTranscriptUsage — single-pass O(n) shape on a 500-line transcript', () => {
// 500 distinct-requestId main-chain assistant records, each input_tokens:10.
const lines = [];
for (let i = 0; i < 500; i++) {
lines.push(JSON.stringify({
type: 'assistant',
isSidechain: false,
requestId: `req_${i}`,
message: { model: 'claude-opus-4-8', usage: { input_tokens: 10, output_tokens: 0, cache_creation_input_tokens: 0, cache_read_input_tokens: 0 } },
}));
}
const totals = parseTranscriptUsage(lines.join('\n'));
assert.equal(totals.tokens_input, 5000); // 500 × 10
});
// ---- deriveCost ------------------------------------------------------------
test('deriveCost — hand-computed value for a known model (cache-aware)', () => {
const totals = { tokens_input: 3000, tokens_output: 500, tokens_cache_creation: 1000, tokens_cache_read: 6000 };
const { cost_usd, is_estimate } = deriveCost(totals, 'claude-opus-4-8');
// (3000×5 + 500×25 + 1000×6.25 + 6000×0.5) / 1e6
// = (15000 + 12500 + 6250 + 3000) / 1e6 = 36750 / 1e6 = 0.03675
assert.equal(cost_usd, 0.03675);
assert.equal(is_estimate, false);
});
test('deriveCost — hand-computed value for claude-fable-5 (v5.9 fable tier)', () => {
const totals = { tokens_input: 3000, tokens_output: 500, tokens_cache_creation: 1000, tokens_cache_read: 6000 };
const { cost_usd, is_estimate } = deriveCost(totals, 'claude-fable-5');
// (3000×10 + 500×50 + 1000×12.5 + 6000×1) / 1e6
// = (30000 + 25000 + 12500 + 6000) / 1e6 = 73500 / 1e6 = 0.0735
assert.equal(cost_usd, 0.0735);
assert.equal(is_estimate, false);
});
test('deriveCost — refuse-to-estimate for an unknown model', () => {
const totals = { tokens_input: 3000, tokens_output: 500, tokens_cache_creation: 1000, tokens_cache_read: 6000 };
const out = deriveCost(totals, 'claude-unknown-9');
assert.equal(out.cost_usd, null);
assert.equal(out.is_estimate, true);
});
test('PRICE_TABLE — frozen, opus-4-8 carries all five rate fields', () => {
assert.ok(Object.isFrozen(PRICE_TABLE), 'PRICE_TABLE must be frozen (drift-pin)');
const p = PRICE_TABLE['claude-opus-4-8'];
for (const k of ['input', 'output', 'cache_read', 'cache_write_5m', 'cache_write_1h']) {
assert.equal(typeof p[k], 'number', `missing rate field ${k}`);
}
});
// ---- buildRecord -----------------------------------------------------------
test('buildRecord — stamps scope:main-context + price_table_version + flat numerics', () => {
const totals = { tokens_input: 3000, tokens_output: 500, tokens_cache_creation: 1000, tokens_cache_read: 6000 };
const rec = buildRecord({ sessionId: 'S1', model: 'claude-opus-4-8', totals, now: '2026-06-26T12:00:00Z' });
assert.equal(rec.scope, 'main-context');
assert.equal(rec.price_table_version, PRICE_TABLE_VERSION);
assert.equal(rec.ts, '2026-06-26T12:00:00Z');
assert.equal(rec.session_id, 'S1');
assert.equal(rec.model, 'claude-opus-4-8');
assert.equal(rec.tokens_input, 3000);
assert.equal(rec.cost_usd, 0.03675);
assert.equal(rec.is_estimate, false);
});
test('buildRecord — unknown model → cost_usd null + is_estimate true (honesty contract)', () => {
const rec = buildRecord({ sessionId: 'S9', model: 'mystery', totals: { tokens_input: 1 }, now: 't' });
assert.equal(rec.cost_usd, null);
assert.equal(rec.is_estimate, true);
});
// ---- upsertSessionRecord ---------------------------------------------------
test('upsertSessionRecord — REPLACES a same-session line (not append)', () => {
const existing =
JSON.stringify({ session_id: 'S1', tokens_input: 1, scope: 'main-context' }) + '\n' +
JSON.stringify({ session_id: 'S2', tokens_input: 2, scope: 'main-context' }) + '\n';
const rec = buildRecord({ sessionId: 'S1', model: 'claude-opus-4-8', totals: { tokens_input: 999 }, now: 't' });
const out = upsertSessionRecord(existing, rec);
const recs = out.trim().split('\n').map(JSON.parse);
assert.equal(recs.length, 2, 'replace must NOT add a line');
const s1 = recs.find(r => r.session_id === 'S1');
assert.equal(s1.tokens_input, 999, 'S1 must carry the new totals');
assert.ok(recs.some(r => r.session_id === 'S2'), 'S2 must be preserved');
});
test('upsertSessionRecord — APPENDS a new-session line', () => {
const existing = JSON.stringify({ session_id: 'S1', tokens_input: 1 }) + '\n';
const rec = buildRecord({ sessionId: 'S3', model: 'claude-opus-4-8', totals: { tokens_input: 7 }, now: 't' });
const out = upsertSessionRecord(existing, rec);
const recs = out.trim().split('\n').map(JSON.parse);
assert.equal(recs.length, 2, 'new session must append');
assert.ok(recs.some(r => r.session_id === 'S3' && r.tokens_input === 7));
});
test('upsertSessionRecord — empty file → single record, trailing newline', () => {
const rec = buildRecord({ sessionId: 'S1', model: 'claude-opus-4-8', totals: { tokens_input: 5 }, now: 't' });
const out = upsertSessionRecord('', rec);
assert.ok(out.endsWith('\n'));
assert.deepEqual(out.trim().split('\n').map(JSON.parse).length, 1);
});
// ---- lastMainChainModel ----------------------------------------------------
test('lastMainChainModel — last main-chain model wins; sidechain excluded even when last', () => {
// Two distinct main-chain models (opus then sonnet) → sonnet (last) wins.
// A sidechain record sits AFTER sonnet: if exclusion broke it would win;
// if last-wins broke, opus (first) would win. Correct answer pins both.
const text = [
JSON.stringify({ type: 'assistant', isSidechain: false, message: { model: 'claude-opus-4-8', usage: {} } }),
JSON.stringify({ type: 'assistant', isSidechain: false, message: { model: 'claude-sonnet-4-6', usage: {} } }),
JSON.stringify({ type: 'assistant', isSidechain: true, message: { model: 'claude-sidechain-ZZZ', usage: {} } }),
JSON.stringify({ type: 'user', message: { content: 'ignored' } }),
].join('\n');
assert.equal(lastMainChainModel(text), 'claude-sonnet-4-6');
});
test('lastMainChainModel — no main-chain model → null → deriveCost refuses to estimate', () => {
// Only a sidechain model, a user line, and a malformed line: no main-chain model.
const text = [
JSON.stringify({ type: 'assistant', isSidechain: true, message: { model: 'claude-opus-4-8', usage: {} } }),
JSON.stringify({ type: 'user', message: { content: 'hi' } }),
'not json',
].join('\n');
const model = lastMainChainModel(text);
assert.equal(model, null);
// null model must propagate to refuse-to-estimate (honesty contract).
const { cost_usd, is_estimate } = deriveCost({ tokens_input: 1000 }, model);
assert.equal(cost_usd, null);
assert.equal(is_estimate, true);
});
test('lastMainChainModel — empty / nullish input → null', () => {
assert.equal(lastMainChainModel(''), null);
assert.equal(lastMainChainModel(null), null);
assert.equal(lastMainChainModel(undefined), null);
});

422
tests/scripts/comparison-gate.test.mjs
View file

@ -1,422 +0,0 @@
// tests/scripts/comparison-gate.test.mjs
// The comparison gate counts the areas where Voyage + the operator's way of working is
// level with or ahead of every reference framework, and fells the table when a claim
// has no evidence behind it. It also counts the "tidy" denominator: every Voyage part a
// documented Claude Code primitive covers, and how many of those are resolved.
//
// Like the yardstick's tests, these pin the MECHANICS in both directions against
// throwaway fixtures. They deliberately do NOT assert that the real table is red today:
// the gate is supposed to turn green, and a test that pins today's distance would fail
// on the day the work is done. Nothing here reads the real table or the real source.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, dirname } from 'node:path';
import { execFileSync } from 'node:child_process';
import {
parseLoc,
parseSourceAreas,
listParts,
evaluate,
render,
main,
} from '../../scripts/comparison-gate.mjs';
// --- helpers ---------------------------------------------------------------
const lines = (n) => `${Array.from({ length: n }, (_, i) => `line ${i + 1}`).join('\n')}\n`;
const SOURCE = [
'# report',
'',
'## 2. Per kapabilitet',
'',
'| Kapabilitet | A | B |',
'|---|---|---|',
'| Alpha | x | y |',
'| Beta | x | y |',
'',
'## 3. Next section',
'',
'| Not | an | area |',
'|---|---|---|',
'| Gamma | x | y |',
'',
].join('\n');
function world() {
const dir = mkdtempSync(join(tmpdir(), 'comparison-gate-'));
const files = {
'vy/own.md': lines(5),
'pm/source.md': SOURCE,
'pm/why.md': lines(3),
'sp/ref.md': lines(10),
};
for (const [rel, body] of Object.entries(files)) {
const p = join(dir, rel);
mkdirSync(dirname(p), { recursive: true });
writeFileSync(p, body);
}
return { dir, roots: { VY: join(dir, 'vy'), PM: join(dir, 'pm'), SP: join(dir, 'sp') } };
}
const cleanup = (w) => rmSync(w.dir, { recursive: true, force: true });
// A table that is fully green: every area level or ahead, every cell with evidence,
// an independent reader, and the one covered part resolved with a measured reason.
function greenData(w) {
return {
schema: 'voyage-comparison/1',
source: { loc: 'PM:source.md', heading: '## 2. Per kapabilitet' },
roots: w.roots,
references: [{ id: 'SP', name: 'obra/superpowers' }],
reader: { kind: 'uavhengig', who: 'fixture', date: '2026-10-02' },
areas: [
{
id: 'alpha',
title: 'Alpha',
own: { evidence: [{ loc: 'VY:own.md:1-2', says: 'own side' }] },
cells: {
SP: { verdict: 'likt', why: 'level', evidence: [{ loc: 'SP:ref.md:3-10', says: 'ref side' }] },
},
},
{
id: 'beta',
title: 'Beta',
own: { evidence: [{ loc: 'VY:own.md:5', says: 'own side' }] },
cells: {
SP: {
verdict: 'foran',
why: 'the reference has nothing here',
evidence: [{
absence: { query: 'grep -rn beta .', hits: 0, known_positive: 'grep -rn line . finds 10' },
says: 'no beta in the reference',
}],
},
},
},
],
tidy: {
primitives: [{ id: 'subagents', name: 'Subagents', doc: 'https://code.claude.com/docs/en/sub-agents', quote: 'q' }],
parts: [
{
path: 'commands/x.md', covered_by: ['subagents'], coverage: 'delvis', why: 'w',
resolution: 'maalt-grunn', reason: { loc: 'PM:why.md:1-3', says: 'measured' },
},
{ path: 'lib/y.mjs', covered_by: [], coverage: 'ikke-dekket', why: 'w', resolution: null, reason: null },
],
},
};
}
const PARTS = ['commands/x.md', 'lib/y.mjs'];
const run = (w, data) => evaluate(data, { parts: PARTS });
const problemText = (r) => r.problems.join('\n');
// --- parsing -----------------------------------------------------------------
test('parseLoc: root, path and a line or a line range', () => {
assert.deepEqual(parseLoc('SP:skills/a/SKILL.md:38-88'), { root: 'SP', path: 'skills/a/SKILL.md', start: 38, end: 88 });
assert.deepEqual(parseLoc('VY:lib/x.mjs:7'), { root: 'VY', path: 'lib/x.mjs', start: 7, end: 7 });
assert.equal(parseLoc('lib/x.mjs:7'), null, 'a loc without a root is not evidence');
assert.equal(parseLoc('VY:lib/x.mjs'), null, 'a loc without a line is not evidence');
assert.equal(parseLoc('VY:lib/x.mjs:9-3'), null, 'a range that runs backwards is not evidence');
assert.equal(parseLoc('VY:lib/x.mjs:0'), null, 'lines count from 1');
});
test('parseSourceAreas: reads the first column of the table under the heading, and only that table', () => {
assert.deepEqual(parseSourceAreas(SOURCE, '## 2. Per kapabilitet'), ['Alpha', 'Beta']);
assert.equal(parseSourceAreas(SOURCE, '## 9. Missing'), null);
});
test('listParts: tracked files in the scope, eval cases as one part each, nothing outside the scope', () => {
const dir = mkdtempSync(join(tmpdir(), 'comparison-gate-parts-'));
try {
for (const rel of ['commands/a.md', 'lib/u/b.mjs', 'evals/case-1/eval.json', 'evals/case-1/fixture/x.md',
'evals/case-2/eval.json', 'docs/not-a-part.md', 'settings.json', 'untracked/z.md']) {
mkdirSync(dirname(join(dir, rel)), { recursive: true });
writeFileSync(join(dir, rel), 'x\n');
}
execFileSync('git', ['init', '-q'], { cwd: dir });
execFileSync('git', ['add', 'commands', 'lib', 'evals', 'docs', 'settings.json'], { cwd: dir });
assert.deepEqual(listParts(dir), ['commands/a.md', 'evals/case-1/', 'evals/case-2/', 'lib/u/b.mjs', 'settings.json']);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
// --- the comparison ------------------------------------------------------------
test('a fully evidenced table, every area level or ahead, read by an independent reader, is green', () => {
const w = world();
try {
const r = run(w, greenData(w));
assert.deepEqual(r.problems, []);
assert.equal(r.comparison.level_or_ahead, 2);
assert.equal(r.comparison.total, 2);
assert.equal(r.green, true);
} finally { cleanup(w); }
});
test('a cell with NO evidence fells the table, whatever its verdict says', () => {
const w = world();
try {
const data = greenData(w);
data.areas[0].cells.SP.evidence = [];
const r = run(w, data);
assert.match(problemText(r), /alpha.*SP.*no evidence/);
assert.equal(r.green, false);
} finally { cleanup(w); }
});
test('an area with no evidence on our own side fells the table', () => {
const w = world();
try {
const data = greenData(w);
delete data.areas[1].own;
const r = run(w, data);
assert.match(problemText(r), /beta.*own.*no evidence/);
assert.equal(r.green, false);
} finally { cleanup(w); }
});
test('a cited line past the end of the file is not evidence', () => {
const w = world();
try {
const data = greenData(w);
data.areas[0].cells.SP.evidence[0].loc = 'SP:ref.md:3-11';
const r = run(w, data);
assert.match(problemText(r), /SP:ref\.md:3-11.*10 lines/);
assert.equal(r.green, false);
} finally { cleanup(w); }
});
test('a cited file that does not exist, or that escapes its root, is not evidence', () => {
const w = world();
try {
const data = greenData(w);
data.areas[0].cells.SP.evidence[0].loc = 'SP:missing.md:1';
data.areas[0].own.evidence[0].loc = 'VY:../pm/source.md:1';
const r = run(w, data);
assert.match(problemText(r), /SP:missing\.md:1.*does not exist/);
assert.match(problemText(r), /VY:\.\.\/pm\/source\.md:1.*outside its root/);
} finally { cleanup(w); }
});
test('an absence without a known-positive control is not evidence, and neither is one with hits', () => {
const w = world();
try {
const data = greenData(w);
delete data.areas[1].cells.SP.evidence[0].absence.known_positive;
let r = run(w, data);
assert.match(problemText(r), /beta.*SP.*known_positive/);
const again = greenData(w);
again.areas[1].cells.SP.evidence[0].absence.hits = 2;
r = run(w, again);
assert.match(problemText(r), /beta.*SP.*hits/);
} finally { cleanup(w); }
});
test('a web citation needs a verbatim quote', () => {
const w = world();
try {
const data = greenData(w);
data.areas[0].cells.SP.evidence = [{ url: 'https://example.org/playbook#review', says: 'x' }];
let r = run(w, data);
assert.match(problemText(r), /alpha.*SP.*quote/);
data.areas[0].cells.SP.evidence = [{ url: 'https://example.org/playbook#review', quote: 'every change is reviewed', says: 'x' }];
r = run(w, data);
assert.deepEqual(r.problems, []);
} finally { cleanup(w); }
});
test('the denominator comes from the SOURCE: an area the source lists but the table drops fells it', () => {
const w = world();
try {
const data = greenData(w);
data.areas.pop();
const r = run(w, data);
assert.match(problemText(r), /Beta.*missing from the table/);
assert.equal(r.comparison.total, 2, 'the total is the source count, not the table count');
assert.equal(r.green, false);
} finally { cleanup(w); }
});
test('an area the source does not list is not counted and is reported', () => {
const w = world();
try {
const data = greenData(w);
data.areas.push({ ...data.areas[0], id: 'gamma', title: 'Gamma' });
const r = run(w, data);
assert.match(problemText(r), /Gamma.*not in the source/);
assert.equal(r.comparison.total, 2);
} finally { cleanup(w); }
});
test('a missing reference cell fells the table; an unknown verdict fells it', () => {
const w = world();
try {
const data = greenData(w);
data.references.push({ id: 'PS', name: 'mattpocock/skills' });
let r = run(w, data);
assert.match(problemText(r), /alpha.*PS.*missing/);
const again = greenData(w);
again.areas[0].cells.SP.verdict = 'probably fine';
r = run(w, again);
assert.match(problemText(r), /alpha.*SP.*verdict/);
} finally { cleanup(w); }
});
test('one reference ahead of us in one area: that area is not counted, and it is named', () => {
const w = world();
try {
const data = greenData(w);
data.areas[0].cells.SP.verdict = 'bak';
const r = run(w, data);
assert.deepEqual(r.problems, []);
assert.equal(r.comparison.level_or_ahead, 1);
assert.deepEqual(r.comparison.behind, [{ area: 'alpha', references: ['SP'] }]);
assert.equal(r.green, false);
} finally { cleanup(w); }
});
test('the count per reference shows which framework pulls the total down', () => {
const w = world();
try {
const data = greenData(w);
data.references.push({ id: 'PS', name: 'mattpocock/skills' });
for (const a of data.areas) a.cells.PS = { ...a.cells.SP };
data.areas[0].cells.PS = { ...data.areas[0].cells.SP, verdict: 'bak' };
const r = run(w, data);
assert.deepEqual(r.problems, []);
assert.deepEqual(r.comparison.per_reference, { SP: 2, PS: 1 });
assert.equal(r.comparison.level_or_ahead, 1);
assert.match(render(r), /PS: 1 of 2/);
} finally { cleanup(w); }
});
test('14 of 14 is not green until an INDEPENDENT reader judged it', () => {
const w = world();
try {
const data = greenData(w);
data.reader.kind = 'pm-rapport';
const r = run(w, data);
assert.deepEqual(r.problems, []);
assert.equal(r.comparison.level_or_ahead, 2);
assert.equal(r.green, false);
assert.match(r.reasons.join('\n'), /independent reader/);
} finally { cleanup(w); }
});
test('a pinned reference whose clone has moved is not evidence for the pinned revision', () => {
const w = world();
try {
execFileSync('git', ['init', '-q'], { cwd: w.roots.SP });
execFileSync('git', ['add', '.'], { cwd: w.roots.SP });
execFileSync('git', ['-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-qm', 'x'], { cwd: w.roots.SP });
const data = greenData(w);
data.references[0].pin = '0000000';
const r = run(w, data);
assert.match(problemText(r), /SP.*pinned 0000000.*HEAD/);
data.references[0].pin = execFileSync('git', ['rev-parse', '--short', 'HEAD'], { cwd: w.roots.SP, encoding: 'utf-8' }).trim();
assert.deepEqual(run(w, data).problems, []);
} finally { cleanup(w); }
});
// --- tidy ------------------------------------------------------------------------
test('tidy: a tracked part the list leaves out fells it; the denominator is the tree, not the list', () => {
const w = world();
try {
const r = evaluate(greenData(w), { parts: [...PARTS, 'agents/new.md'] });
assert.match(problemText(r), /agents\/new\.md.*missing from the tidy list/);
assert.equal(r.tidy.parts, 3);
assert.equal(r.green, false);
} finally { cleanup(w); }
});
test('tidy: a covered part counts as resolved only with a measured reason or as removed from the tree', () => {
const w = world();
try {
const data = greenData(w);
data.tidy.parts[0].resolution = 'aapen';
data.tidy.parts[0].reason = null;
let r = run(w, data);
assert.deepEqual(r.problems, []);
assert.equal(r.tidy.covered, 1);
assert.equal(r.tidy.resolved, 0);
assert.equal(r.green, false);
const noReason = greenData(w);
noReason.tidy.parts[0].reason = null;
r = run(w, noReason);
assert.match(problemText(r), /commands\/x\.md.*maalt-grunn.*reason/);
const removed = greenData(w);
removed.tidy.parts[0] = { ...removed.tidy.parts[0], resolution: 'fjernet', reason: null, commit: 'abc1234' };
r = run(w, removed);
assert.match(problemText(r), /commands\/x\.md.*fjernet.*still in the tree/);
r = evaluate(removed, { parts: ['lib/y.mjs'] });
assert.deepEqual(r.problems, []);
assert.equal(r.tidy.resolved, 1);
} finally { cleanup(w); }
});
test('tidy: a primitive without a documentation URL, or a part covered by an unknown primitive, fells it', () => {
const w = world();
try {
const data = greenData(w);
data.tidy.primitives[0].doc = 'somewhere';
data.tidy.parts[0].covered_by = ['subagents', 'nonexistent'];
const r = run(w, data);
assert.match(problemText(r), /subagents.*documentation URL/);
assert.match(problemText(r), /commands\/x\.md.*nonexistent/);
} finally { cleanup(w); }
});
// --- output and CLI ------------------------------------------------------------------
test('render names the count, the behind areas and the reader', () => {
const w = world();
try {
const data = greenData(w);
data.areas[0].cells.SP.verdict = 'bak';
data.reader.kind = 'pm-rapport';
const text = render(run(w, data));
assert.match(text, /1 of 2/);
assert.match(text, /alpha \(SP\)/);
assert.match(text, /pm-rapport/);
assert.match(text, /RED/);
} finally { cleanup(w); }
});
test('main: a missing table is NOT MEASURED and exits 1, never 0; an unknown flag exits 2', () => {
const w = world();
try {
assert.equal(main(['--data', join(w.dir, 'nope.json'), '--json']), 1);
assert.equal(main(['--nope']), 2);
} finally { cleanup(w); }
});
test('main: exit 0 only for a green table, 1 for a red one', () => {
const w = world();
try {
const file = join(w.dir, 'table.json');
writeFileSync(file, JSON.stringify(greenData(w)));
// --parts-from pins the denominator to a fixture repo instead of the real tree
const repo = join(w.dir, 'repo');
for (const rel of PARTS) {
mkdirSync(dirname(join(repo, rel)), { recursive: true });
writeFileSync(join(repo, rel), 'x\n');
}
execFileSync('git', ['init', '-q'], { cwd: repo });
execFileSync('git', ['add', '.'], { cwd: repo });
assert.equal(main(['--data', file, '--parts-from', repo, '--json']), 0);
const red = greenData(w);
red.areas[0].cells.SP.evidence = [];
writeFileSync(file, JSON.stringify(red));
assert.equal(main(['--data', file, '--parts-from', repo, '--json']), 1);
} finally { cleanup(w); }
});

950
tests/scripts/end-state-gate.test.mjs
View file

@ -1,950 +0,0 @@
// tests/scripts/end-state-gate.test.mjs
// The end-state gate counts the distance to "Voyage is finished" as five tallies:
// open pipeline defects, unresolved experiments/opt-ins, dormant agents, open operator
// decisions, and feat commits after the freeze tag. "Finished" = all five at 0 AND the
// registry the gate counts from is intact against its frozen denominator.
//
// These tests pin the gate's MECHANICS in both directions — every tally can be 0 and can
// be > 0 — against throwaway fixture trees. They deliberately do NOT assert that the real
// repo is red today: the gate is supposed to go green, and a test that pins the current
// distance would fail on the day the work is done. What they DO pin about the real repo
// is the denominator itself (see "the frozen denominator" below).
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync, existsSync, readFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, dirname, resolve } from 'node:path';
import { spawnSync } from 'node:child_process';
import { fileURLToPath } from 'node:url';
import {
evaluateCondition,
evaluateCheck,
measure,
loadRegistry,
loadFrozen,
spawnedNames,
checkSignature,
verifyIntegrity,
render,
PROBE_NOTES,
} from '../../scripts/end-state-gate.mjs';
const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..', '..');
const GATE = join(ROOT, 'scripts', 'end-state-gate.mjs');
// --- the frozen denominator ------------------------------------------------
// Denominator frozen 2026-09-17, the day the end-state
// direction was chosen. An entry may CLOSE; it may never be removed or have its check
// changed, and the gate's agent/decision/freeze configuration may not move. To change
// anything below you must change this test — say in the commit message who decided it.
const FROZEN = {
defects: {
'D-01': '48f7f5cca070091e8f9b9ca0c7cdb7a1880733691935c25d07ea1d3e8d508fc9',
'D-02': 'dfc94dba04a9116ae9be2097a0c5a9d8313b02c7de1ca6b49a472c1823a6d1e1',
// Amended 2026-09-18 (operator decision): D-03/D-04 became behaviour probes.
// Amended a second time the same day, as a maintenance decision: those
// behaviour probes were greps for an export name, and a two-line stub
// exporting both names closed them on an unfixed tree (mutant M6, pinned
// below). They now RUN a named test.
// Amended a third time the same day, same standing: a named test can be
// EMPTY, and two empty tests with the right names closed both defects on a
// tree without lib/verification/ (mutant M7, pinned below). Each condition
// now also names the module whose STUB its test must fell. Every amendment
// moved a signature on purpose and removed nothing.
'D-03': '98698ce74c243353713786b8604160f6672568b090884e9428d9e5e9fbcad89d',
'D-04': 'df11a2e95db7b014ce077db0473e5dde4195ace92d8077cacdc1cee273947b47',
'D-05': '8fe6df52b43496c3302400507cb98002dcf8d2a12d92085e14bda4c7819bc7ba',
'D-06': '72a787c8154d1c18849e9856dcf1a67703d6b5541f88db81d6bab13cab14b418',
'D-07': '1bf3e0cd36c621720697475b0a3fd7db78611d66b58a9ce035e7695e57c69f1b',
},
experiments: {
'E-01': 'f7718897cec8ace6879fe79f5f32760ec957c3642e7186f8aa1688ce77b3f4e8',
'E-02': 'c8ff12c504ba7a9ac36f11bb8f9b713263e9b4eb28c07d984d9e2115980e740d',
'E-03': '17153ad8808615a833a49ae0c3523264ce055aa690e5542a75bae1874495e7e8',
},
agents: {
dir: 'agents',
spawnSites: 'commands/*.md',
referenceMarker: 'Reference document, not a spawnable capability',
},
decisions: {
file: 'STATE.md',
section: '## Åpne operatørbeslutninger',
open: '^- \\[ \\] ',
closed: '^- \\[x\\] ',
},
freeze: { tag: 'end-state-freeze', featPattern: '^feat(\\(|!|:)' },
};
// --- helpers ---------------------------------------------------------------
function fixture(files) {
const dir = mkdtempSync(join(tmpdir(), 'end-state-gate-'));
for (const [rel, body] of Object.entries(files)) {
const p = join(dir, rel);
mkdirSync(dirname(p), { recursive: true });
writeFileSync(p, body);
}
return dir;
}
const cleanup = (...dirs) => { for (const d of dirs) rmSync(d, { recursive: true, force: true }); };
const DECISIONS = { file: 'STATE.md', section: '## Open decisions', open: '^- \\[ \\] ', closed: '^- \\[x\\] ' };
const AGENTS = { dir: 'agents', spawnSites: 'commands/*.md', referenceMarker: 'Reference document, not a spawnable capability' };
const FREEZE = { tag: 'end-state-freeze', featPattern: '^feat(\\(|!|:)' };
function registry(overrides = {}) {
return { defects: [], experiments: [], agents: AGENTS, decisions: DECISIONS, freeze: FREEZE, ...overrides };
}
// A frozen manifest that matches `reg` exactly.
function frozenFor(reg) {
const sigs = (list) => Object.fromEntries(list.map((e) => [e.id, checkSignature(e.check)]));
return { defects: sigs(reg.defects), experiments: sigs(reg.experiments), agents: reg.agents, decisions: reg.decisions, freeze: reg.freeze };
}
function git(dir, ...args) {
const r = spawnSync('git', [
'-c', 'user.name=fixture', '-c', 'user.email=fixture@example.invalid',
'-c', 'core.hooksPath=/dev/null', '-c', 'commit.gpgsign=false', '-c', 'tag.gpgsign=false',
...args,
], { cwd: dir, encoding: 'utf8' });
assert.equal(r.status, 0, `git ${args.join(' ')}: ${r.stderr}`);
return r.stdout;
}
function gitInit(dir) {
git(dir, 'init', '-q');
git(dir, 'add', '-A');
git(dir, 'commit', '-q', '--allow-empty', '-m', 'chore: fixture baseline');
}
function gitCommit(dir, subject) {
git(dir, 'commit', '-q', '--allow-empty', '-m', subject);
}
const rowById = (result, id) => result.rows.find((r) => r.id === id);
// --- conditions -----------------------------------------------------------
test('evaluateCondition: match / no-match on a single file, multiline regex', () => {
const dir = fixture({ 'a.md': 'intro\n### Heading\nbody\n' });
try {
assert.equal(evaluateCondition(dir, { path: 'a.md', pattern: '^### Heading', flags: 'm', expect: 'match' }), true);
assert.equal(evaluateCondition(dir, { path: 'a.md', pattern: '^### Other', flags: 'm', expect: 'match' }), false);
assert.equal(evaluateCondition(dir, { path: 'a.md', pattern: '^### Other', flags: 'm', expect: 'no-match' }), true);
} finally { cleanup(dir); }
});
test('evaluateCondition: globs (dir/*.ext, dir/**/*.ext) and path arrays match ANY file', () => {
const dir = fixture({
'commands/x.md': 'nothing here\n',
'lib/deep/y.mjs': 'const FLAG = process.env.SOME_FLAG;\n',
});
try {
assert.equal(evaluateCondition(dir, { path: 'commands/*.md', pattern: 'SOME_FLAG', expect: 'match' }), false);
assert.equal(evaluateCondition(dir, { path: 'lib/**/*.mjs', pattern: 'SOME_FLAG', expect: 'match' }), true);
assert.equal(evaluateCondition(dir, { path: ['commands/*.md', 'lib/**/*.mjs'], pattern: 'SOME_FLAG', expect: 'match' }), true);
assert.equal(evaluateCondition(dir, { path: ['commands/*.md', 'lib/**/*.mjs'], pattern: 'SOME_FLAG', expect: 'no-match' }), false);
} finally { cleanup(dir); }
});
test('evaluateCondition: section scoping reads only from the heading to the next "## "', () => {
const dir = fixture({
'doc.md': '## 5. Design\nno status here\n## 6. PoC\n> **STATUS: RUN AND DECLINED**\n',
});
try {
const inFive = { path: 'doc.md', section: '## 5.', pattern: '^> \\*\\*STATUS:', flags: 'm', expect: 'no-match' };
const inSix = { ...inFive, section: '## 6.' };
assert.equal(evaluateCondition(dir, inFive), true, 'section 5 has no status block');
assert.equal(evaluateCondition(dir, inSix), false, 'section 6 has one');
} finally { cleanup(dir); }
});
test('evaluateCondition: a literal NUL byte is detectable', () => {
const dir = fixture({ 'nul.mjs': "const SEP = '\u0000';\n", 'clean.mjs': "const SEP = '\\x00';\n" });
try {
assert.equal(evaluateCondition(dir, { path: 'nul.mjs', pattern: '\\u0000', expect: 'match' }), true);
assert.equal(evaluateCondition(dir, { path: 'clean.mjs', pattern: '\\u0000', expect: 'match' }), false);
} finally { cleanup(dir); }
});
test('evaluateCondition: a missing file, a missing glob base or a missing section throws', () => {
const dir = fixture({ 'a.md': 'x\n' });
try {
assert.throws(() => evaluateCondition(dir, { path: 'missing.md', pattern: 'x', expect: 'no-match' }), /missing\.md/);
assert.throws(() => evaluateCondition(dir, { path: 'nodir/*.md', pattern: 'x', expect: 'no-match' }), /nodir/);
assert.throws(() => evaluateCondition(dir, { path: 'a.md', section: '## 9.', pattern: 'x', expect: 'no-match' }), /## 9\./);
} finally { cleanup(dir); }
});
test('evaluateCondition: a glob over an EXISTING dir with no matching files throws (no silent no-match)', () => {
// In a no-match condition, "silently nothing matched" would read as the condition holding.
const dir = fixture({ 'docs/readme.txt': 'x\n' });
try {
assert.throws(() => evaluateCondition(dir, { path: 'docs/*.md', pattern: 'x', expect: 'no-match' }), /docs\/\*\.md/);
assert.throws(() => evaluateCondition(dir, { path: 'docs/**/*.md', pattern: 'x', expect: 'no-match' }), /docs\/\*\*\/\*\.md/);
} finally { cleanup(dir); }
});
test('evaluateCheck: open only when ALL conditions hold; null check is not fellable', () => {
const dir = fixture({
'r.md': 'run the `### Bridge agent` block\n',
'fixed.md': 'run the `### Bridge agent` block\n### Bridge agent\n',
});
const dangling = (file) => [
{ path: file, pattern: '`### Bridge agent`', expect: 'match' },
{ path: file, pattern: '^### Bridge agent', flags: 'm', expect: 'no-match' },
];
try {
assert.equal(evaluateCheck(dir, dangling('r.md')).status, 'open');
assert.equal(evaluateCheck(dir, dangling('fixed.md')).status, 'closed');
assert.equal(evaluateCheck(dir, null).status, 'not-fellable');
const broken = evaluateCheck(dir, dangling('gone.md'));
assert.equal(broken.status, 'not-fellable');
assert.match(broken.detail, /gone\.md/);
} finally { cleanup(dir); }
});
// --- spawn instructions (what "referenced by a command" means) -------------
test('spawnedNames: a row of a table headed `Agent` is a spawn instruction; other tables are not', () => {
const md = [
'Launch the new agents:', '',
'| Agent | Purpose |', '|-------|---------|', '| `alpha-agent` | does a |', '| `beta-agent` | does b |', '',
'| Name | Purpose |', '|------|---------|', '| `gamma-agent` | just a list |', '',
].join('\n');
const names = spawnedNames(md);
assert.ok(names.has('alpha-agent') && names.has('beta-agent'));
assert.ok(!names.has('gamma-agent'), 'a table not headed "Agent" is not a roster');
});
test('spawnedNames: an imperative Launch/Spawn line names the agents it launches', () => {
const md = [
'Launch the **alpha-agent** agent:',
'- Launch `beta-agent` (Agent tool) with the findings.',
'3. Spawn `voyage:gamma-agent` in the background.',
'**Launch** the `delta-agent` agent now.',
'launch `epsilon-agent` too',
].join('\n');
const names = spawnedNames(md);
for (const n of ['alpha-agent', 'beta-agent', 'gamma-agent', 'delta-agent', 'epsilon-agent']) {
assert.ok(names.has(n), `${n} should count as spawned`);
}
});
test('spawnedNames: an agent block with a Prompt: line is a spawn instruction; a bare bold name is not', () => {
const md = [
'**alpha-agent** — adversarial review of the plan.',
'Prompt: "Review this plan."',
'',
'**beta-agent** — mentioned for context only.',
'It is not launched here.',
].join('\n');
const names = spawnedNames(md);
assert.ok(names.has('alpha-agent'));
assert.ok(!names.has('beta-agent'));
});
test('spawnedNames: prose, negation, comments, fences and quotes are never spawn instructions', () => {
const md = [
'The alpha-agent is dormant and is never spawned.',
'Never launch `beta-agent`.',
'In quick mode we used to launch `gamma-agent`.',
'<!-- Launch the `delta-agent` agent -->',
'<!--',
'Launch the `epsilon-agent` agent',
'-->',
'```',
'Launch the `zeta-agent` agent',
'| Agent |',
'|---|',
'| `eta-agent` |',
'```',
'> Launch the `theta-agent` agent',
].join('\n');
const names = spawnedNames(md);
for (const n of ['alpha-agent', 'beta-agent', 'gamma-agent', 'delta-agent', 'epsilon-agent', 'zeta-agent', 'eta-agent', 'theta-agent']) {
assert.ok(!names.has(n), `${n} must not count as spawned`);
}
});
// --- the tallies, both directions ------------------------------------------
test('defects tally: > 0 while a check holds, 0 once fixed; not-fellable counts as open', () => {
const defect = {
id: 'D-X',
summary: 'removed tool still referenced',
probe: 'phrase',
check: [{ path: 'commands/plan.md', pattern: '\\bTeamCreate\\b', expect: 'match' }],
};
const unfellable = { id: 'D-Y', summary: 'no runnable check yet', check: null };
const base = { 'STATE.md': '## Open decisions\n', 'agents/a.md': 'x\n', 'commands/plan.md': 'Launch `a`\n' };
const red = fixture({ ...base, 'commands/plan.md': 'Launch `a` and TeamCreate\n' });
const green = fixture(base);
try {
let row = rowById(measure(red, registry({ defects: [defect] })), 'defects');
assert.equal(row.open, 1);
assert.equal(row.total, 1);
row = rowById(measure(green, registry({ defects: [defect] })), 'defects');
assert.equal(row.open, 0);
row = rowById(measure(green, registry({ defects: [defect, unfellable] })), 'defects');
assert.equal(row.open, 1, 'a defect without a runnable check stays open');
assert.equal(row.items.find((i) => i.id === 'D-Y').status, 'not-fellable');
} finally { cleanup(red, green); }
});
test('experiments tally: an env-gated opt-in is open until its gate is gone', () => {
const exp = {
id: 'E-X',
summary: 'default-off loop',
probe: 'phrase',
check: [{ path: ['commands/*.md', 'lib/**/*.mjs'], pattern: 'LOOP_ENABLED', expect: 'match' }],
};
const base = { 'STATE.md': '## Open decisions\n', 'agents/a.md': 'x\n', 'commands/r.md': 'Launch `a`\n' };
const red = fixture({ ...base, 'lib/util/cap.mjs': 'if (process.env.LOOP_ENABLED) {}\n' });
const green = fixture({ ...base, 'lib/util/cap.mjs': 'export const x = 1;\n' });
try {
assert.equal(rowById(measure(red, registry({ experiments: [exp] })), 'experiments').open, 1);
assert.equal(rowById(measure(green, registry({ experiments: [exp] })), 'experiments').open, 0);
} finally { cleanup(red, green); }
});
test('dormant-agents tally: an agent no command spawns counts; reference docs never do', () => {
const files = {
'STATE.md': '## Open decisions\n',
'agents/used-agent.md': '---\nname: used-agent\ndescription: |\n does work\n---\n',
'agents/idle-agent.md': '---\nname: idle-agent\ndescription: |\n never wired\n---\n',
'agents/some-orchestrator.md': '---\nname: some-orchestrator\ndescription: Reference document, not a spawnable capability — docs\n---\n',
'commands/run.md': 'Launch the **used-agent** agent.\n',
};
const red = fixture(files);
const green = fixture({ ...files, 'commands/other.md': 'Spawn `idle-agent` in parallel.\n' });
try {
let row = rowById(measure(red, registry()), 'dormant-agents');
assert.equal(row.open, 1);
assert.equal(row.total, 2, 'denominator = spawnable agents (reference docs excluded)');
assert.deepEqual(row.items.filter((i) => i.status === 'open').map((i) => i.id), ['idle-agent']);
row = rowById(measure(green, registry()), 'dormant-agents');
assert.equal(row.open, 0);
} finally { cleanup(red, green); }
});
test('dormant-agents tally: a prose sentence or an HTML comment does not wake a dormant agent', () => {
const files = {
'STATE.md': '## Open decisions\n',
'agents/idle-agent.md': '---\nname: idle-agent\ndescription: |\n never wired\n---\n',
'commands/plan.md': 'The idle-agent is dormant and is never spawned.\n',
'commands/end.md': '<!-- idle-agent -->\nThe research-idle-agents phase.\n',
};
const dir = fixture(files);
try {
assert.equal(rowById(measure(dir, registry()), 'dormant-agents').open, 1);
} finally { cleanup(dir); }
});
test('dormant-agents tally: an unmeasurable agents dir is n/a (null), never 0', () => {
const dir = fixture({ 'STATE.md': '## Open decisions\n', 'commands/r.md': 'x\n' });
try {
const result = measure(dir, registry());
const row = rowById(result, 'dormant-agents');
assert.equal(row.open, null);
assert.equal(row.total, null);
assert.match(row.detail, /not measurable/);
assert.equal(result.green, false);
} finally { cleanup(dir); }
});
test('decisions tally: counts open markers inside the section only, and says it counts ticks', () => {
const state = [
'# STATE',
'- [ ] outside the section, ignored',
'## Open decisions',
'- [ ] first open',
'- [x] already decided',
'- [ ] second open',
'## Next section',
'- [ ] also ignored',
'',
].join('\n');
const red = fixture({ 'STATE.md': state, 'agents/a.md': 'x\n', 'commands/r.md': 'Launch `a`\n' });
const green = fixture({
'STATE.md': '## Open decisions\n- [x] done\n## Next\n',
'agents/a.md': 'x\n',
'commands/r.md': 'Launch `a`\n',
});
try {
let row = rowById(measure(red, registry()), 'decisions');
assert.equal(row.open, 2);
assert.equal(row.total, 3);
assert.match(row.detail, /tick/i, 'the output must say a tick is not verified as a decision');
row = rowById(measure(green, registry()), 'decisions');
assert.equal(row.open, 0);
assert.equal(row.total, 1);
} finally { cleanup(red, green); }
});
test('decisions tally: a missing STATE.md or section is NOT MEASURABLE (null), never 0', () => {
const noState = fixture({ 'agents/a.md': 'x\n', 'commands/r.md': 'Launch `a`\n' });
const noSection = fixture({ 'STATE.md': '# STATE\n- [ ] x\n', 'agents/a.md': 'x\n', 'commands/r.md': 'Launch `a`\n' });
try {
for (const dir of [noState, noSection]) {
const result = measure(dir, registry());
const row = rowById(result, 'decisions');
assert.equal(row.open, null);
assert.ok(row.detail.length > 0, 'the reason is reported');
assert.equal(result.green, false, 'an unmeasurable tally keeps the gate red');
}
} finally { cleanup(noState, noSection); }
});
test('decisions tally: an unmarked list item in the section is format drift → NOT MEASURABLE', () => {
// A STATE written without the marker convention must not read as "0 open".
for (const item of ['1. **Some decision?** prose', '- plain bullet decision', '* star bullet']) {
const dir = fixture({
'STATE.md': `## Open decisions\n- [ ] marked\n${item}\n## Next\n`,
'agents/a.md': 'x\n',
'commands/r.md': 'Launch `a`\n',
});
try {
const row = rowById(measure(dir, registry()), 'decisions');
assert.equal(row.open, null, `unmarked "${item}" must make the tally unmeasurable`);
assert.match(row.detail, /unmarked/);
} finally { cleanup(dir); }
}
});
test('feat-after-freeze tally: not a git repo → n/a (red), never 0', () => {
const dir = fixture({ 'STATE.md': '## Open decisions\n' });
try {
const result = measure(dir, registry());
const row = rowById(result, 'feat-after-freeze');
assert.equal(row.open, null);
assert.match(row.detail, /not measured/);
assert.equal(result.green, false);
} finally { cleanup(dir); }
});
test('feat-after-freeze tally: no freeze tag yet → n/a (red), and the detail names the tag', () => {
const dir = fixture({ 'STATE.md': '## Open decisions\n' });
try {
gitInit(dir);
const row = rowById(measure(dir, registry()), 'feat-after-freeze');
assert.equal(row.open, null);
assert.match(row.detail, /end-state-freeze/);
} finally { cleanup(dir); }
});
test('feat-after-freeze tally: counts feat commits after the tag, 0 when there are none', () => {
const dir = fixture({ 'STATE.md': '## Open decisions\n' });
try {
gitInit(dir);
gitCommit(dir, 'feat: before the freeze does not count');
git(dir, 'tag', 'end-state-freeze');
let row = rowById(measure(dir, registry()), 'feat-after-freeze');
assert.equal(row.open, 0);
assert.equal(row.total, 0);
gitCommit(dir, 'fix(x): a defect fix is allowed');
gitCommit(dir, 'feature: not a conventional feat');
row = rowById(measure(dir, registry()), 'feat-after-freeze');
assert.equal(row.open, 0);
assert.equal(row.total, 2);
gitCommit(dir, 'feat(research): a new capability');
gitCommit(dir, 'feat!: a breaking capability');
row = rowById(measure(dir, registry()), 'feat-after-freeze');
assert.equal(row.open, 2);
assert.equal(row.total, 4);
} finally { cleanup(dir); }
});
// --- registry integrity (the judged party must not own the ledger) ---------
const SAMPLE = registry({
defects: [
{ id: 'D-01', summary: 'first defect', probe: 'phrase', check: [{ path: 'a.md', pattern: 'bad', expect: 'match' }] },
{ id: 'D-02', summary: 'second defect', probe: 'phrase', check: [{ path: 'a.md', pattern: 'worse', expect: 'match' }] },
],
experiments: [
{ id: 'E-01', summary: 'first experiment', probe: 'phrase', check: [{ path: 'a.md', pattern: 'FLAG', expect: 'match' }] },
{ id: 'E-02', summary: 'second experiment', probe: 'phrase', check: [{ path: 'a.md', pattern: 'OPT', expect: 'match' }] },
],
});
const clone = (v) => JSON.parse(JSON.stringify(v));
test('verifyIntegrity: the frozen registry itself is intact; a new entry may be added', () => {
const frozen = frozenFor(SAMPLE);
assert.deepEqual(verifyIntegrity(SAMPLE, frozen), { ok: true, violations: [] });
const grown = clone(SAMPLE);
grown.defects.push({ id: 'D-03', summary: 'found later', probe: 'phrase', check: [{ path: 'a.md', pattern: 'new', expect: 'match' }] });
assert.equal(verifyIntegrity(grown, frozen).ok, true);
});
test('verifyIntegrity: every registry attack is a violation that names what moved', () => {
const frozen = frozenFor(SAMPLE);
const attacks = {
'delete a defect': (r) => { r.defects = r.defects.filter((d) => d.id !== 'D-02'); },
'delete an experiment': (r) => { r.experiments = r.experiments.filter((e) => e.id !== 'E-01'); },
'weaken a defect check': (r) => { r.defects[0].check[0].pattern = '^NEVER MATCHES$'; },
'weaken an experiment check': (r) => { r.experiments[1].check[0].expect = 'no-match'; },
'drop a condition': (r) => { r.defects[0].check = []; },
'point spawnSites at agents': (r) => { r.agents = { ...r.agents, spawnSites: 'agents/*.md' }; },
'change the reference marker': (r) => { r.agents = { ...r.agents, referenceMarker: 'DORMANT' }; },
'move the decisions section': (r) => { r.decisions = { ...r.decisions, section: '## Elsewhere' }; },
'change the open marker': (r) => { r.decisions = { ...r.decisions, open: '^NEVER' }; },
'rename the freeze tag': (r) => { r.freeze = { ...r.freeze, tag: 'some-future-tag' }; },
'duplicate an id': (r) => { r.defects.push(clone(r.defects[0])); },
};
for (const [name, attack] of Object.entries(attacks)) {
const reg = clone(SAMPLE);
attack(reg);
const v = verifyIntegrity(reg, frozen);
assert.equal(v.ok, false, `${name} must be caught`);
assert.ok(v.violations.length > 0 && v.violations.every((s) => typeof s === 'string' && s.length > 0), name);
}
});
test('verifyIntegrity: spawnSites may never include agents/, even if the manifest says so', () => {
const reg = clone(SAMPLE);
reg.agents = { ...reg.agents, spawnSites: 'agents/*.md' };
const v = verifyIntegrity(reg, frozenFor(reg));
assert.equal(v.ok, false);
assert.ok(v.violations.some((s) => /agents\//.test(s)));
});
test('checkSignature: key order does not matter; every field of a condition does', () => {
const a = [{ path: 'a.md', pattern: 'x', flags: 'm', expect: 'match' }];
const reordered = [{ expect: 'match', flags: 'm', pattern: 'x', path: 'a.md' }];
assert.equal(checkSignature(a), checkSignature(reordered));
for (const change of [{ path: 'b.md' }, { pattern: 'y' }, { flags: '' }, { expect: 'no-match' }, { section: '## 1.' }]) {
assert.notEqual(checkSignature([{ ...a[0], ...change }]), checkSignature(a), JSON.stringify(change));
}
assert.match(checkSignature(a), /^[0-9a-f]{64}$/);
});
test('measure: a missing frozen manifest makes integrity n/a and keeps the gate red', () => {
const dir = fixture({ 'STATE.md': '## Open decisions\n' });
try {
const result = measure(dir, registry());
assert.equal(result.integrity.ok, null);
assert.match(result.integrity.detail, /end-state-frozen\.json/);
assert.equal(result.green, false);
} finally { cleanup(dir); }
});
// --- render ----------------------------------------------------------------
test('render: the header counts only tallies that ARE 0, never n/a ones', () => {
const result = {
green: false,
integrity: { ok: true, violations: [], detail: '' },
rows: [
{ id: 'defects', label: 'l', source: 's', open: 0, total: 1, target: 0, items: [], detail: '' },
{ id: 'experiments', label: 'l', source: 's', open: null, total: null, target: 0, items: [], detail: 'not measurable: x' },
{ id: 'dormant-agents', label: 'l', source: 's', open: 0, total: 3, target: 0, items: [], detail: '' },
],
};
const text = render(result);
assert.match(text, /RED \(2 of 3 tallies at 0\)/);
assert.match(text, /\| experiments \| n\/a \| n\/a \|/);
});
test('render: open phrase probes are labelled, closed ids are listed, integrity is shown', () => {
const result = {
green: false,
integrity: { ok: false, violations: ['D-02 removed from the registry'], detail: '' },
rows: [{
id: 'defects', label: 'open pipeline defects', source: 's', open: 1, total: 2, target: 0, detail: '',
items: [
{ id: 'D-01', summary: 'still open', probe: 'phrase', status: 'open', detail: '' },
{ id: 'D-06', summary: 'fixed', probe: 'byte', status: 'closed', detail: '' },
],
}],
};
const text = render(result);
assert.match(text, /\[open · phrase probe\] D-01/);
assert.match(text, /closed: D-06/);
assert.match(text, /registry integrity: VIOLATED/);
assert.match(text, /D-02 removed from the registry/);
});
// --- CLI -------------------------------------------------------------------
function writeGateFiles(dir, reg, frozen = frozenFor(reg)) {
mkdirSync(join(dir, 'scripts'), { recursive: true });
writeFileSync(join(dir, 'scripts', 'end-state-registry.json'), JSON.stringify(reg, null, 2));
mkdirSync(join(dir, 'tests', 'fixtures'), { recursive: true });
writeFileSync(join(dir, 'tests', 'fixtures', 'end-state-frozen.json'), JSON.stringify(frozen, null, 2));
}
function runGate(dir, extra = []) {
return spawnSync(process.execPath, [GATE, '--root', dir, ...extra], { encoding: 'utf8' });
}
const ALL_IDS = ['defects', 'experiments', 'dormant-agents', 'decisions', 'feat-after-freeze'];
test('CLI: all five tallies at 0 with an intact registry → green, exit 0; --json carries the rows', () => {
const dir = fixture({
'STATE.md': '## Open decisions\n- [x] done\n',
'agents/a.md': '---\nname: a\n---\n',
'commands/r.md': 'Spawn `a`\n',
});
try {
writeGateFiles(dir, registry());
gitInit(dir);
git(dir, 'tag', 'end-state-freeze');
const r = runGate(dir, ['--json']);
assert.equal(r.status, 0, r.stdout + r.stderr);
const out = JSON.parse(r.stdout);
assert.equal(out.green, true);
assert.equal(out.integrity.ok, true);
assert.deepEqual(out.rows.map((x) => x.id), ALL_IDS);
assert.ok(out.rows.every((x) => x.open === 0 && x.target === 0 && typeof x.source === 'string'));
} finally { cleanup(dir); }
});
test('CLI: an intact registry with every measurable tally at 0 but no freeze tag → red, exit 1', () => {
// n/a must never be read as 0 — also when nothing else is wrong.
const dir = fixture({
'STATE.md': '## Open decisions\n- [x] done\n',
'agents/a.md': '---\nname: a\n---\n',
'commands/r.md': 'Spawn `a`\n',
});
try {
writeGateFiles(dir, registry());
gitInit(dir);
const r = runGate(dir, ['--json']);
assert.equal(r.status, 1, r.stdout + r.stderr);
const out = JSON.parse(r.stdout);
assert.equal(out.integrity.ok, true);
assert.equal(out.green, false);
assert.equal(out.rows.find((x) => x.id === 'feat-after-freeze').open, null);
assert.ok(out.rows.filter((x) => x.id !== 'feat-after-freeze').every((x) => x.open === 0));
} finally { cleanup(dir); }
});
test('CLI: all tallies at 0 but a tampered registry → red, exit 1 (the combined green attack)', () => {
const dir = fixture({
'STATE.md': '## Open decisions\n- [x] ticked\n',
'agents/a.md': '---\nname: a\n---\n',
'commands/r.md': 'Spawn `a`\n',
});
try {
const honest = registry({ defects: [{ id: 'D-01', summary: 'a real defect', probe: 'phrase', check: [{ path: 'commands/r.md', pattern: 'Spawn', expect: 'match' }] }] });
const tampered = registry(); // the open defect simply left the ledger
writeGateFiles(dir, tampered, frozenFor(honest));
gitInit(dir);
git(dir, 'tag', 'end-state-freeze');
const r = runGate(dir);
assert.equal(r.status, 1, r.stdout + r.stderr);
assert.match(r.stdout, /RED/);
assert.match(r.stdout, /VIOLATED/);
assert.match(r.stdout, /D-01/);
} finally { cleanup(dir); }
});
test('CLI: any tally > 0 → red, exit 1; the table names every row', () => {
const dir = fixture({
'STATE.md': '## Open decisions\n- [ ] undecided\n',
'agents/a.md': '---\nname: a\n---\n',
'commands/r.md': 'Spawn `a`\n',
});
try {
writeGateFiles(dir, registry());
const r = runGate(dir);
assert.equal(r.status, 1, r.stderr);
for (const label of ALL_IDS) assert.match(r.stdout, new RegExp(label));
assert.match(r.stdout, /RED/);
} finally { cleanup(dir); }
});
test('CLI: an unreadable registry is a usage error (exit 2), not a verdict', () => {
const dir = fixture({ 'STATE.md': '## Open decisions\n' });
try {
const r = runGate(dir);
assert.equal(r.status, 2);
assert.match(r.stderr, /end-state-registry\.json/);
} finally { cleanup(dir); }
});
// --- the real repo: denominator pinned, state never pinned -----------------
test('the frozen denominator: tests/fixtures/end-state-frozen.json is exactly the literal above', () => {
const file = loadFrozen(ROOT);
const { why, ...rest } = file;
assert.ok(typeof why === 'string' && why.length > 20, 'the manifest says why it is frozen');
assert.deepEqual(rest, FROZEN);
});
test('the frozen denominator: the real registry is intact against it', () => {
const v = verifyIntegrity(loadRegistry(ROOT), FROZEN);
assert.deepEqual(v, { ok: true, violations: [] }, v.violations.join('\n'));
});
test('real registry: every counted entry has an id, a summary, a probe kind and a check that RUNS here', () => {
const reg = loadRegistry(ROOT);
const entries = [...reg.defects, ...reg.experiments];
const ids = new Set();
for (const e of entries) {
assert.match(e.id, /^[DE]-\d{2}$/, `bad id ${e.id}`);
assert.ok(!ids.has(e.id), `duplicate id ${e.id}`);
ids.add(e.id);
assert.ok(typeof e.summary === 'string' && e.summary.length > 10, `${e.id} needs a summary`);
assert.ok(Object.keys(PROBE_NOTES).includes(e.probe), `${e.id} must declare a probe kind the gate can explain`);
// A check that cannot evaluate against this very repo is a typo, not a finding.
assert.notEqual(evaluateCheck(ROOT, e.check).status, 'not-fellable', `${e.id} check does not run here`);
}
});
test('real repo: five rows, open is a count or null, exit code agrees with rows + integrity', () => {
const r = spawnSync(process.execPath, [GATE, '--json'], { encoding: 'utf8', cwd: ROOT });
assert.ok(r.status === 0 || r.status === 1, r.stderr);
const out = JSON.parse(r.stdout);
assert.deepEqual(out.rows.map((x) => x.id), ALL_IDS);
for (const row of out.rows) {
assert.ok(row.open === null || (Number.isInteger(row.open) && row.open >= 0), `${row.id}: ${row.open}`);
}
const green = out.rows.every((row) => row.open === 0) && out.integrity.ok === true;
assert.equal(out.green, green);
assert.equal(r.status, green ? 0 : 1);
assert.ok(existsSync(join(ROOT, 'scripts', 'end-state-registry.json')));
assert.ok(readFileSync(join(ROOT, 'tests', 'fixtures', 'end-state-frozen.json'), 'utf8').length > 0);
});
// --- probe labelling -------------------------------------------------------
//
// The gate labelled only `phrase` probes, so every other probe kind rendered
// unlabelled — indistinguishable from an entry that declared nothing. Once
// D-03/D-04 became behaviour probes (operator decision 2026-09-18) the reader
// needs to see WHICH kind of evidence each closed entry rests on.
test('PROBE_NOTES: every probe kind the registry may declare carries a note', () => {
assert.deepEqual(Object.keys(PROBE_NOTES).sort(), ['behaviour', 'byte', 'phrase']);
for (const [kind, note] of Object.entries(PROBE_NOTES)) {
assert.ok(note.length > 20, `${kind} needs a note that says what the probe can and cannot prove`);
}
});
test('render: every probe kind is labelled, open and closed — not just phrase', () => {
const text = render({
green: false,
integrity: { ok: true, violations: [], detail: '' },
rows: [{
id: 'defects', label: 'open pipeline defects', source: 's', open: 2, total: 3, target: 0, detail: '',
items: [
{ id: 'D-01', summary: 'still open', probe: 'phrase', status: 'open', detail: '' },
{ id: 'D-03', summary: 'still open', probe: 'behaviour', status: 'open', detail: '' },
{ id: 'D-06', summary: 'fixed', probe: 'byte', status: 'closed', detail: '' },
],
}],
});
assert.match(text, /\[open · phrase probe\] D-01/);
assert.match(text, /\[open · behaviour probe\] D-03/);
assert.match(text, /closed: D-06 \(byte probe\)/);
});
test('the detail line carries one note per probe kind PRESENT, and none for kinds absent', () => {
const dir = fixture({
'a.md': 'bad\n',
'STATE.md': '## Open decisions\n- [x] done\n',
'agents/a.md': '---\nname: a\n---\n',
'commands/r.md': 'Spawn `a`\n',
});
try {
writeGateFiles(dir, registry({
defects: [
{ id: 'D-01', summary: 'a phrase defect', probe: 'phrase', check: [{ path: 'a.md', pattern: 'bad', expect: 'match' }] },
{ id: 'D-02', summary: 'a behaviour defect', probe: 'behaviour', check: [{ path: 'a.md', pattern: 'nope', expect: 'match' }] },
],
}));
gitInit(dir);
const out = JSON.parse(runGate(dir, ['--json']).stdout);
const detail = rowById(out, 'defects').detail;
assert.match(detail, /phrase probe closes on rewording/);
assert.match(detail, /behaviour probe/);
assert.ok(!/byte probe/.test(detail), 'a kind no entry declares must not be explained');
assert.equal(rowById(out, 'experiments').detail, '', 'an empty row explains nothing');
} finally { cleanup(dir); }
});
// --- a probe that RUNS a test, and the M6 mutant ---------------------------
//
// Measured 2026-09-18 (re-measured on 6cafb4c): a two-line file
// export function runPlanVerification() { return { ok: true }; }
// export function formatCriteriaEvidence() { return ''; }
// dropped into an otherwise UNFIXED tree closed both D-03 and D-04, and the
// gate reported "defects 0 of 7, registry intact". A grep condition can only
// ask whether a string is present, so a behaviour probe built from one is a
// phrase probe pointed at code. A condition that RUNS a named test and reads
// its TAP result cannot be closed that way.
const M6_STUB = [
'export function runPlanVerification() { return { ok: true }; }',
"export function formatCriteriaEvidence() { return ''; }",
'',
].join('\n');
// A fixture pair: a module, and a test that BINDS it. `green one` passes on
// the tree and fails against a stub of the module - which is what makes it a
// behaviour probe rather than a name.
const MODULE = 'export function answer() { return 42; }\n';
const TEST_FILE = [
"import { test } from 'node:test';",
"import { strict as assert } from 'node:assert';",
"import { answer } from '../lib/m.mjs';",
"test('green one', () => { assert.equal(answer(), 42); });",
"test('red one', () => { assert.ok(false, 'red on purpose'); });",
'',
].join('\n');
test('a test condition RUNS the named test and reads its result', () => {
const dir = fixture({ 'tests/t.test.mjs': TEST_FILE, 'lib/m.mjs': MODULE });
try {
const cond = (name, expect) => ({ test: 'tests/t.test.mjs', name, expect, stubs: ['lib/m.mjs'] });
assert.equal(evaluateCondition(dir, cond('green one', 'passes')), true);
assert.equal(evaluateCondition(dir, cond('green one', 'fails')), false);
assert.equal(evaluateCondition(dir, cond('red one', 'passes')), false);
assert.equal(evaluateCondition(dir, cond('red one', 'fails')), true);
} finally { cleanup(dir); }
});
test('a test condition naming a test that does not run is NOT FELLABLE, never closed', () => {
const dir = fixture({ 'tests/t.test.mjs': TEST_FILE, 'lib/m.mjs': MODULE });
try {
for (const cond of [
{ test: 'tests/t.test.mjs', name: 'no such test', expect: 'fails', stubs: ['lib/m.mjs'] },
{ test: 'tests/missing.test.mjs', name: 'green one', expect: 'fails', stubs: ['lib/m.mjs'] },
]) {
const r = evaluateCheck(dir, [cond]);
assert.equal(r.status, 'not-fellable', JSON.stringify(cond));
assert.ok(r.detail.length > 0, 'the gate must say WHY the check could not fire');
}
} finally { cleanup(dir); }
});
test('M6 mutant: a stub exporting both symbols does NOT close D-03 or D-04', () => {
const dir = fixture({ 'lib/cosmetic/stub.mjs': M6_STUB });
try {
const entries = loadRegistry(ROOT).defects.filter((e) => e.id === 'D-03' || e.id === 'D-04');
assert.equal(entries.length, 2, 'D-03 and D-04 are still in the registry');
for (const e of entries) {
assert.notEqual(
evaluateCheck(dir, e.check).status, 'closed',
`${e.id} closed on a tree where nothing is fixed: ${JSON.stringify(e.check)}`,
);
}
} finally { cleanup(dir); }
});
test('the gate states out loud that the wiring is pinned by text, not proven', () => {
const r = spawnSync(process.execPath, [GATE], { encoding: 'utf8', cwd: ROOT });
assert.ok(r.status === 0 || r.status === 1, r.stderr);
assert.match(r.stdout, /wiring: /);
assert.match(r.stdout, /pinned by TEXT/);
assert.match(r.stdout, /plugin-eval/);
});
// --- M7: a probe must bind the test's CONTENT, not its name ---------------
//
// Measured 2026-09-18 (re-measured on e1e7bdf): the cosmetic close came back
// one level up. The probe reads the test file from the SAME tree it measures,
// so a four-line file holding two EMPTY tests with the two named names —
// test('runPlanVerification: a plan whose success criterion FAILS fells the run', () => {});
// test('formatCriteriaEvidence: one row per criterion, with command and exit code', () => {});
// — closed D-03 and D-04 on a tree where `lib/verification/` did not exist at
// all: "defects 0 of 7, registry intact". An empty test is the new two-line
// stub. The only form that cannot be closed by a name is one that requires the
// named test to FELL a mutant: the gate runs it a second time against a tree
// where the module it binds is replaced by a stub exporting the same names and
// doing nothing. A test that passes against both is not a behaviour probe.
const EMPTY_NAMED_TESTS = [
"import test from 'node:test';",
"test('runPlanVerification: a plan whose success criterion FAILS fells the run', () => {});",
"test('formatCriteriaEvidence: one row per criterion, with command and exit code', () => {});",
'',
].join('\n');
const REAL_RUNNER = join(ROOT, 'lib', 'verification', 'criteria-runner.mjs');
const REAL_RUNNER_TEST = join(ROOT, 'tests', 'lib', 'criteria-runner.test.mjs');
test('M7 mutant: two EMPTY tests with the right names do NOT close D-03 or D-04', () => {
// The checkpoint's reproduction: an unfixed tree (no lib/verification/) plus
// the four-line test file.
const dir = fixture({ 'tests/lib/criteria-runner.test.mjs': EMPTY_NAMED_TESTS });
try {
const entries = loadRegistry(ROOT).defects.filter((e) => e.id === 'D-03' || e.id === 'D-04');
assert.equal(entries.length, 2, 'D-03 and D-04 are still in the registry');
for (const e of entries) {
const r = evaluateCheck(dir, e.check);
assert.equal(r.status, 'not-fellable', `${e.id} closed on a tree where nothing is fixed`);
assert.ok(r.detail.length > 0, 'the gate must say WHY the check could not fire');
}
} finally { cleanup(dir); }
});
test('M7 mutant: an empty named test does not close the defect even when the module IS present', () => {
// The harder case: the real module is there, so the stub CAN be built — and
// the empty test passes against it, which is exactly the proof that the test
// binds the name and nothing else.
const dir = fixture({
'tests/lib/criteria-runner.test.mjs': EMPTY_NAMED_TESTS,
'lib/verification/criteria-runner.mjs': readFileSync(REAL_RUNNER, 'utf8'),
});
try {
for (const e of loadRegistry(ROOT).defects.filter((x) => x.id === 'D-03' || x.id === 'D-04')) {
const r = evaluateCheck(dir, e.check);
assert.equal(r.status, 'not-fellable', `${e.id} closed on an empty test`);
assert.match(r.detail, /stub/i, 'the gate must name the mutant the test failed to fell');
}
} finally { cleanup(dir); }
});
test("D-03/D-04 still CLOSE on the real tree — the named tests do fell a stub", () => {
// The positive control. Without it, "not closed" everywhere would read as a
// working probe when the probe is simply broken.
for (const e of loadRegistry(ROOT).defects.filter((x) => x.id === 'D-03' || x.id === 'D-04')) {
const r = evaluateCheck(ROOT, e.check);
assert.equal(r.status, 'closed', `${e.id} is not closed on the real tree: ${r.detail}`);
}
});
test('a test condition must name the module its test binds, or it cannot fire', () => {
const dir = fixture({ 'tests/t.test.mjs': TEST_FILE, 'lib/m.mjs': MODULE });
try {
for (const cond of [
{ test: 'tests/t.test.mjs', name: 'green one', expect: 'passes' },
{ test: 'tests/t.test.mjs', name: 'green one', expect: 'passes', stubs: [] },
{ test: 'tests/t.test.mjs', name: 'green one', expect: 'passes', stubs: ['lib/gone.mjs'] },
]) {
const r = evaluateCheck(dir, [cond]);
assert.equal(r.status, 'not-fellable', JSON.stringify(cond));
assert.ok(r.detail.length > 0, 'the gate must say WHY');
}
} finally { cleanup(dir); }
});
test('the stub mutant leaves the measured tree untouched', () => {
const dir = fixture({ 'tests/t.test.mjs': TEST_FILE, 'lib/m.mjs': MODULE });
try {
const before = readFileSync(join(dir, 'lib', 'm.mjs'), 'utf8');
evaluateCheck(dir, [{ test: 'tests/t.test.mjs', name: 'green one', expect: 'passes', stubs: ['lib/m.mjs'] }]);
assert.equal(readFileSync(join(dir, 'lib', 'm.mjs'), 'utf8'), before, 'the mutation must happen in a copy, never in place');
} finally { cleanup(dir); }
});
test('the frozen file claims no authority it does not have', () => {
// Measured the same day: the tracked frozen file claimed its second
// amendment was operator-authorised. No operator authorised it; it followed
// from a maintenance decision. A file whose whole job is to be an explicit
// decision trail may not overclaim in either direction.
const why = loadFrozen(ROOT).why;
assert.ok(why.length > 0, 'the frozen manifest must say WHY it changed');
assert.ok(
!/operator-authoris|operator-authoriz|by work order|PM checkpoint/i.test(why),
'the frozen file may not claim operator authorisation, and coordination metadata belongs in the local plan',
);
});

4
tests/scripts/storm-measure.test.mjs
View file

@ -13,7 +13,7 @@
// - a stats file with no `effort` field is a loud error, never a silently
// empty group that reads as "no gain".
//
// Pattern: tests/scripts/synthesis-measure.test.mjs (pure core, no fixtures on disk).
// Pure core, no fixtures on disk.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
@ -248,7 +248,7 @@ test('decideVerdict: a null gain is insufficient data, never a decline', () => {
// appends; nothing read it.
//
// The stats record carries counts, not names — names are free prose and
// field-allowlist.mjs denies prose by omission — so the run attests membership
// the record denies prose by omission — so the run attests membership
// with a low-cardinality boolean instead, and the gate refuses to call
// activation OK without it.
// ---------------------------------------------------------------------------

101
tests/scripts/synthesis-measure.test.mjs
View file

@ -1,101 +0,0 @@
// tests/scripts/synthesis-measure.test.mjs
// NW3 (S12) — deterministic Δ main-context measurement core.
//
// The gate metric (T1 §2) is Δ main-context tokens for an equivalent-quality
// digest. This pins the pure accounting that turns fixture token counts into a
// verdict, under the two framings that decide NW3:
//
// - FAITHFUL (current flow): Phase 5 swarm runs FOREGROUND, so its outputs are
// already RESIDENT in main before Phase 7. Delegating only the synthesis read
// cannot evict them → main holds base+out+dig in BOTH arms → Δ ≈ 0.
// - DISK-POTENTIAL (upper bound): IF outputs were on disk (a separate Phase-5
// change, out of NW3 scope), the delegated arm holds base+dig only → Δ = out/(base+out+dig).
//
// POSITIVE adopt requires Δ ≥ 30% AND quality ≥ inline (T1 §5 thresholds).
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import {
estimateTokens,
mainContextTokens,
deltaPct,
decideVerdict,
analyze,
} from '../../scripts/synthesis-measure.mjs';
test('estimateTokens: ~chars/4 heuristic, monotonic, non-negative', () => {
assert.equal(estimateTokens(''), 0);
assert.equal(estimateTokens('abcd'), 1);
assert.equal(estimateTokens('abcde'), 2); // ceil(5/4)
assert.ok(estimateTokens('a'.repeat(400)) === 100);
assert.ok(estimateTokens('x'.repeat(1000)) > estimateTokens('x'.repeat(500)));
});
test('mainContextTokens: inline arm holds base + out + dig', () => {
assert.equal(mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'inline' }), 63500);
});
test('mainContextTokens: delegated_faithful still holds out (Phase-5 resident) + dig', () => {
// The decisive structural fact: foreground swarm delivery already made `out`
// resident; delegating Phase 7 does not evict it.
assert.equal(
mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'delegated_faithful' }),
63500,
);
});
test('mainContextTokens: delegated_disk holds base + dig only (out lives in the sub-agent)', () => {
assert.equal(
mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'delegated_disk' }),
51500,
);
});
test('mainContextTokens: unknown arm throws (no silent default)', () => {
assert.throws(() => mainContextTokens({ base: 1, out: 1, dig: 1, arm: 'nope' }));
});
test('deltaPct: (A-B)/A; equal arms → 0; B smaller → positive fraction', () => {
assert.equal(deltaPct(100, 100), 0);
assert.equal(deltaPct(100, 75), 0.25);
assert.equal(deltaPct(0, 0), 0); // guard divide-by-zero
});
test('decideVerdict: ≥30% AND quality-ok → POSITIVE', () => {
assert.equal(decideVerdict(0.30, true), 'POSITIVE');
assert.equal(decideVerdict(0.45, true), 'POSITIVE');
});
test('decideVerdict: quality loss forces NEGATIVE even at a large Δ', () => {
assert.equal(decideVerdict(0.60, false), 'NEGATIVE');
});
test('decideVerdict: <15% → NEGATIVE; the [15%,30%) band → INCONCLUSIVE', () => {
assert.equal(decideVerdict(0.149, true), 'NEGATIVE');
assert.equal(decideVerdict(0.00, true), 'NEGATIVE');
assert.equal(decideVerdict(0.15, true), 'INCONCLUSIVE');
assert.equal(decideVerdict(0.2999, true), 'INCONCLUSIVE');
});
test('analyze: faithful arm is structurally Δ=0 → NEGATIVE regardless of sizes', () => {
const a = analyze({ baseTokens: 50000, outTokens: 12000, digTokens: 1500, qualityOK: true });
assert.equal(a.faithful.deltaPct, 0);
assert.equal(a.faithful.verdict, 'NEGATIVE');
assert.equal(a.faithful.armA, a.faithful.armB);
});
test('analyze: disk arm Δ = out/(base+out+dig)', () => {
const a = analyze({ baseTokens: 50000, outTokens: 12000, digTokens: 1500, qualityOK: true });
const expected = 12000 / (50000 + 12000 + 1500);
assert.ok(Math.abs(a.disk.deltaPct - expected) < 1e-9);
assert.equal(a.disk.armB, 51500);
});
test('analyze: disk verdict tracks BASE — large fixed baseline can sink the upper bound below 30%', () => {
// Small base → disk Δ clears 30%; large base → it does not. Demonstrates the
// upper bound is itself BASE-sensitive (sweep, do not assert one number).
const small = analyze({ baseTokens: 10000, outTokens: 12000, digTokens: 1500, qualityOK: true });
const large = analyze({ baseTokens: 200000, outTokens: 12000, digTokens: 1500, qualityOK: true });
assert.equal(small.disk.verdict, 'POSITIVE'); // 12000/23500 ≈ 0.51
assert.equal(large.disk.verdict, 'NEGATIVE'); // 12000/213500 ≈ 0.056
});

660
tests/scripts/yardstick.test.mjs
View file

@ -1,660 +0,0 @@
// tests/scripts/yardstick.test.mjs
// The yardstick prints Voyage's three numbers (use · trust · interrupts) with
// denominators read from their SOURCE, plus a fourth row that asks whether the
// end-state gate itself can be fooled by a signature-preserving mutant.
//
// Like the gate's own test, these tests pin the MECHANICS in both directions
// against throwaway fixtures - every number can be countable and can be not
// countable, the mutant row can read felled and not felled. They deliberately
// do NOT assert that the real data is red today: the yardstick is supposed to
// turn green, and a test that pins today's distance would fail on the day the
// work is done. Nothing here reads the real order queue or the real stats.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync, readFileSync, symlinkSync } from 'node:fs';
import { spawnSync } from 'node:child_process';
import { tmpdir } from 'node:os';
import { join, dirname, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import {
DEFAULT_CUTOFF,
MARKER_SINCE,
loadStats,
readOrders,
measureUse,
measureTrust,
measureInterrupts,
emitterCallSites,
buildM9Tree,
measureMutants,
measureAll,
render,
main,
} from '../../scripts/yardstick.mjs';
// New exports are read through the namespace, so a missing one fails its own
// tests instead of failing this whole file at link time.
import * as Y from '../../scripts/yardstick.mjs';
const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..', '..');
// --- helpers ---------------------------------------------------------------
function fixture(files) {
const dir = mkdtempSync(join(tmpdir(), 'yardstick-'));
for (const [rel, body] of Object.entries(files)) {
const p = join(dir, rel);
mkdirSync(dirname(p), { recursive: true });
writeFileSync(p, body);
}
return dir;
}
const cleanup = (...dirs) => { for (const d of dirs) rmSync(d, { recursive: true, force: true }); };
const jsonl = (...rows) => `${rows.map((r) => (typeof r === 'string' ? r : JSON.stringify(r))).join('\n')}\n`;
// `marker` writes the order's solution-choice marker the way dispatch does: an
// HTML comment on its own line in the body, `<!-- loesningsvalg: <value>; grunn=... -->`.
// `arbeidsform` writes `<!-- arbeidsform: <value> -->` on its own line, as dispatch does.
function order({ id, date, workClass, slug, marker, arbeidsform, extra = '' }) {
const lines = ['---', 'from: .claude', 'to: some-repo', `order-id: ${id}`, `subject: order ${id}`, `date: ${date}`];
if (workClass !== undefined) lines.push(`work-class: ${workClass}`);
if (slug !== undefined) lines.push(`slug: ${slug}`);
lines.push('---');
if (arbeidsform !== undefined) lines.push(`<!-- arbeidsform: ${arbeidsform} -->`);
if (marker !== undefined) lines.push(`<!-- loesningsvalg: ${marker}; grunn=fixture -->`);
lines.push('', `# ORDER ${id}`, extra, '');
return lines.join('\n');
}
const review = (ts, verdict, slug = 's') => ({ ts, slug, verdict });
// A /trekbrief stats record naming the order it ran (the writer adds order_*).
const briefRun = ({ order_id, slug = 's', ts = '2026-10-04T10:00:00Z' }) => ({ ts, slug, order_id, order_claimed_at: ts, order_reason: order_id ? null : 'none-claimed' });
// The runs index the yardstick joins on, built from stats files `{ name: [records] }`.
function runsFrom(files = {}) {
const dir = fixture(Object.fromEntries(Object.entries(files).map(([n, recs]) => [n, jsonl(...recs)])));
try { return Y.orderRuns(loadStats(dir)); } finally { cleanup(dir); }
}
const runsOf = (...records) => runsFrom(records.length ? { 'trekbrief-stats.jsonl': records } : {});
// measureUse over a fixture queue; counting starts at MARKER_SINCE unless the test says otherwise.
function use(files, { records = [], runs, cutoff = DEFAULT_CUTOFF, countingSince = MARKER_SINCE } = {}) {
const coord = fixture(files);
try {
return measureUse({ queue: readOrders(coord), runs: runs ?? runsOf(...records), cutoff, countingSince });
} finally { cleanup(coord); }
}
// --- the stats loader --------------------------------------------------------
test('loadStats: an invalid JSON line is COUNTED, never silently dropped', () => {
const dir = fixture({
'trekreview-stats.jsonl': jsonl(review('2026-09-01T00:00:00Z', 'ALLOW'), '{not json', '', review('2026-09-02T00:00:00Z', 'BLOCK')),
});
try {
const stats = loadStats(dir);
const f = stats.files.find((x) => x.name === 'trekreview-stats.jsonl');
assert.equal(f.lines, 3, 'blank lines are not lines; the invalid one is');
assert.equal(f.records.length, 2);
assert.deepEqual(f.invalid, [2], 'the invalid line is reported by its line number');
} finally { cleanup(dir); }
});
test('loadStats: a missing data directory is reported, not read as zero', () => {
const stats = loadStats(join(tmpdir(), 'yardstick-no-such-dir-8f3a'));
assert.equal(stats.missing, true);
assert.deepEqual(stats.files, []);
});
// --- 2 · TRUST ---------------------------------------------------------------
test('trust: the ALLOW streak is read in ts order, not file order', () => {
// File order ends on ALLOW, ALLOW; chronologically the last review is a BLOCK.
const dir = fixture({
'trekreview-stats.jsonl': jsonl(
review('2026-09-01T00:00:00Z', 'ALLOW'),
review('2026-09-05T00:00:00Z', 'BLOCK'),
review('2026-09-02T00:00:00Z', 'ALLOW'),
review('2026-09-03T00:00:00Z', 'ALLOW'),
),
});
try {
const row = measureTrust(loadStats(dir));
assert.equal(row.countable, true);
assert.equal(row.denominator, 4);
assert.equal(row.value, 0, 'the chronologically last review is a BLOCK');
assert.equal(row.facts.longestStreak, 3);
assert.deepEqual(row.facts.verdicts, { ALLOW: 3, BLOCK: 1 });
assert.equal(row.facts.last.verdict, 'BLOCK');
} finally { cleanup(dir); }
});
test('trust: both spacings of the verdict field count the same (it is read as JSON, not grepped)', () => {
const dir = fixture({
'trekreview-stats.jsonl': jsonl(
'{"ts": "2026-09-01T00:00:00Z", "slug": "a", "verdict": "BLOCK"}',
'{"ts":"2026-09-02T00:00:00Z","slug":"b","verdict":"ALLOW"}',
),
});
try {
const row = measureTrust(loadStats(dir));
assert.deepEqual(row.facts.verdicts, { BLOCK: 1, ALLOW: 1 });
assert.equal(row.value, 1);
} finally { cleanup(dir); }
});
test('trust: an invalid line is left out of n and said out loud', () => {
const dir = fixture({
'trekreview-stats.jsonl': jsonl(review('2026-09-01T00:00:00Z', 'ALLOW'), '{"ts": oops'),
});
try {
const row = measureTrust(loadStats(dir));
assert.equal(row.denominator, 1);
assert.equal(row.facts.invalid, 1);
assert.match(row.detail, /1 invalid line/);
} finally { cleanup(dir); }
});
test('trust: an empty or missing review file is NOT countable', () => {
const empty = fixture({ 'trekreview-stats.jsonl': '' });
const none = fixture({ 'trekplan-stats.jsonl': jsonl({ ts: '2026-09-01T00:00:00Z', slug: 'x' }) });
try {
for (const dir of [empty, none]) {
const row = measureTrust(loadStats(dir));
assert.equal(row.countable, false, dir);
assert.equal(row.value, null);
assert.ok(row.detail.length > 0, 'the row must say WHY it cannot be counted');
}
} finally { cleanup(empty, none); }
});
// --- 1 · USE -----------------------------------------------------------------
test('use: orders are read from orders/, archive/ and claimed/, and nothing else', () => {
const coord = fixture({
'repo-a/orders/o1.md': order({ id: 'o1', date: '2026-09-22T10:00:00Z' }),
'repo-a/orders/archive/o2.md': order({ id: 'o2', date: '2026-09-01T10:00:00Z' }),
'repo-b/orders/claimed/o3.md': order({ id: 'o3', date: '2026-09-22T10:00:00Z' }),
'repo-b/orders/claimed/o3.claim': 'claimed',
'repo-b/inbox/m1.md': order({ id: 'm1', date: '2026-09-22T10:00:00Z' }),
'repo-b/orders/broken.md': 'no frontmatter at all\n',
});
try {
const q = readOrders(coord);
assert.equal(q.files, 4, 'three orders and one unreadable file, all .md under */orders/');
assert.equal(q.orders.length, 3);
assert.equal(q.unreadable.length, 1);
} finally { cleanup(coord); }
});
test('use: a dot-mailbox is a mailbox - its orders count, and the detail names it', () => {
// Measured 2026-09-22: a hand count over `coord/*/orders` came out lower than
// the yardstick, because the shell's `*` skips dot-directories and the queue
// does not. The detail names them, so the next re-measure can reconcile.
const coord = fixture({
'r/orders/o1.md': order({ id: 'o1', date: '2026-09-22T10:00:00Z' }),
'.pm/orders/archive/o2.md': order({ id: 'o2', date: '2026-09-22T10:00:00Z' }),
});
try {
const q = readOrders(coord);
assert.equal(q.orders.length, 2);
const row = measureUse({ queue: q, runs: runsOf(), cutoff: DEFAULT_CUTOFF, countingSince: MARKER_SINCE });
assert.match(row.detail, /2 order files in 2 mailboxes/);
assert.match(row.detail, /dot-mailboxes .pm 1/);
} finally { cleanup(coord); }
});
test('use: the marker is counted from 2026-09-28, the day pre-order-loesningsvalg.sh began enforcing it', () => {
assert.equal(MARKER_SINCE, '2026-09-28');
});
test('use: no order carrying a loesningsvalg marker means NOT COUNTABLE, not 0 %', () => {
const row = use({
'r/orders/o1.md': order({ id: 'o1', date: '2026-09-29T10:00:00Z' }),
'r/orders/o2.md': order({ id: 'o2', date: '2026-09-30T10:00:00Z' }),
});
assert.equal(row.countable, false);
assert.equal(row.value, null);
assert.equal(row.denominator, 0);
assert.equal(row.facts.scanned, 2);
assert.equal(row.facts.withMarker, 0);
assert.equal(row.facts.notMeasured, 2, 'an unmarked order is not measured - never a no');
assert.match(row.detail, /0 of 2 orders carry a loesningsvalg marker/);
assert.match(row.detail, /2 not measured/);
});
test('use: plan and prove are NEW work, ikke-nytt is not; unmarked and pre-2026-09-28 orders are not measured', () => {
const row = use({
'r/orders/plan.md': order({ id: 'plan', date: '2026-09-28T00:00:00Z', marker: 'plan' }),
'r/orders/archive/prove.md': order({ id: 'prove', date: '2026-10-01T00:00:00Z', marker: 'prove' }),
'r/orders/claimed/ikke.md': order({ id: 'ikke', date: '2026-10-01T00:00:00Z', marker: 'ikke-nytt' }),
'r/orders/none.md': order({ id: 'none', date: '2026-10-01T00:00:00Z' }),
'r/orders/early.md': order({ id: 'early', date: '2026-09-27T23:59:59Z', marker: 'plan' }),
}, { records: [briefRun({ order_id: 'plan' }), briefRun({ order_id: 'ikke', slug: 'other' }), briefRun({ order_id: 'early', slug: 'old' })] });
assert.equal(row.facts.newWork, 2, 'plan and prove');
assert.equal(row.facts.notNew, 1);
assert.equal(row.facts.beforeMarker, 1);
assert.equal(row.facts.unmarked, 1);
assert.equal(row.facts.withVoyageRun, 1, 'plan has a trekbrief record naming it');
assert.equal(row.facts.noRecord, 1, 'prove has none - not measured, never a no');
assert.equal(row.countable, true);
assert.equal(row.value, 1);
assert.equal(row.denominator, 1, 'YES + declared NO; prove is not measured');
assert.equal(row.facts.notMeasured, 3, 'unmarked, before 2026-09-28, and prove without a record');
assert.match(row.detail, /value = 1 of 1 measured/);
});
test('use: work-class in the frontmatter no longer decides anything - only the marker does', () => {
const row = use({
'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', workClass: 'new' }),
'r/orders/b.md': order({ id: 'b', date: '2026-09-29T00:00:00Z', workClass: 'maintenance', marker: 'prove' }),
}, { records: [briefRun({ order_id: 'a' }), briefRun({ order_id: 'b', slug: 'b' })] });
assert.equal(row.facts.newWork, 1, 'b is new by its marker; a has no marker and is not measured');
assert.equal(row.value, 1);
assert.equal(row.denominator, 1);
assert.equal(row.facts.notMeasured, 1);
});
test('use: only a marker on its own line counts - a mention in prose or an unknown value is not measured', () => {
const row = use({
'r/orders/prose.md': order({ id: 'prose', date: '2026-09-29T00:00:00Z', extra: 'Count from the marker `<!-- loesningsvalg: plan -->` in the body.' }),
'r/orders/odd.md': order({ id: 'odd', date: '2026-09-29T00:00:00Z', marker: 'kanskje' }),
'r/orders/both.md': order({ id: 'both', date: '2026-09-29T00:00:00Z', marker: 'ikke-nytt', extra: 'Prose names `<!-- loesningsvalg: plan -->` too.' }),
});
assert.equal(row.denominator, 0);
assert.equal(row.facts.notNew, 1, 'both: the line marker wins over the prose mention');
assert.equal(row.facts.notMeasured, 2, 'prose has no marker; kanskje is no known value');
assert.equal(row.countable, false);
});
test('use: a cutoff later than 2026-09-28 moves the start of the count', () => {
const row = use({
'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', marker: 'plan' }),
'r/orders/b.md': order({ id: 'b', date: '2026-10-02T00:00:00Z', marker: 'plan' }),
}, { cutoff: '2026-10-01', records: [briefRun({ order_id: 'a' }), briefRun({ order_id: 'b', slug: 'b' })] });
assert.equal(row.facts.newWork, 1);
assert.equal(row.facts.beforeMarker, 1);
assert.equal(row.value, 1, 'b only');
assert.match(row.source, /from 2026-10-01/);
});
// --- use: the order-id join (source C) -------------------------------------------
test('use: a new-work order is YES only when a trekbrief record names its order id', () => {
const row = use({ 'r/orders/o.md': order({ id: 'o', date: '2026-09-29T00:00:00Z', marker: 'plan' }) },
{ records: [briefRun({ order_id: 'o' })] });
assert.equal(row.countable, true);
assert.equal(row.value, 1);
assert.equal(row.denominator, 1);
assert.equal(row.facts.withVoyageRun, 1);
assert.deepEqual(row.facts.matched, ['o']);
});
test('use: no record, or a record naming another id, is NOT MEASURED - never a no', () => {
const row = use({ 'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', marker: 'plan' }) },
{ records: [briefRun({ order_id: 'zzz' })] });
assert.equal(row.facts.noRecord, 1);
assert.equal(row.facts.withVoyageRun, 0);
assert.equal(row.countable, false);
assert.equal(row.value, null);
assert.equal(row.denominator, 0);
assert.equal(row.facts.notMeasured, 1);
});
test('use: an equal slug alone no longer matches - only the order id does', () => {
const row = use({ 'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', marker: 'plan', slug: 'feat-a' }) },
{ records: [briefRun({ order_id: null, slug: 'feat-a' }), briefRun({ order_id: 'other', slug: 'feat-a' })] });
assert.equal(row.facts.withVoyageRun, 0);
assert.equal(row.facts.noRecord, 1);
});
test('use: an order id in another stats file does not count - only /trekbrief records the order', () => {
const row = use({ 'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', marker: 'plan' }) },
{ runs: runsFrom({ 'trekplan-stats.jsonl': [{ ts: 't', slug: 's', order_id: 'a' }] }) });
assert.equal(row.facts.withVoyageRun, 0);
assert.equal(row.facts.noRecord, 1);
});
test('use: two records with the same id and slug count once; the same id under two slugs is disputed', () => {
const once = use({ 'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', marker: 'plan' }) },
{ records: [briefRun({ order_id: 'a' }), briefRun({ order_id: 'a', ts: '2026-10-05T00:00:00Z' })] });
assert.equal(once.value, 1);
assert.equal(once.denominator, 1);
const two = use({ 'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', marker: 'plan' }) },
{ records: [briefRun({ order_id: 'a', slug: 'x' }), briefRun({ order_id: 'a', slug: 'y' })] });
assert.equal(two.facts.disputed, 1);
assert.equal(two.facts.withVoyageRun, 0);
assert.equal(two.countable, false, 'a disputed order is not measured');
});
test('use: one slug naming two order ids makes both disputed - a brief cannot run two orders', () => {
const row = use({
'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', marker: 'plan' }),
'r/orders/b.md': order({ id: 'b', date: '2026-09-29T00:00:00Z', marker: 'plan' }),
}, { records: [briefRun({ order_id: 'a', slug: 'one' }), briefRun({ order_id: 'b', slug: 'one' })] });
assert.equal(row.facts.disputed, 2);
assert.equal(row.facts.withVoyageRun, 0);
});
test('use: arbeidsform: ordre is a declared NO even with a record; arbeidsform: voyage with a record is YES', () => {
const row = use({
'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', marker: 'plan', arbeidsform: 'ordre' }),
'r/orders/b.md': order({ id: 'b', date: '2026-09-29T00:00:00Z', marker: 'plan', arbeidsform: 'voyage' }),
}, { records: [briefRun({ order_id: 'a', slug: 'a' }), briefRun({ order_id: 'b', slug: 'b' })] });
assert.equal(row.facts.declaredNo, 1);
assert.equal(row.facts.withVoyageRun, 1);
assert.deepEqual(row.facts.matched, ['b']);
assert.equal(row.value, 1);
assert.equal(row.denominator, 2);
});
test('use: a veto marker mentioned mid-line in prose does not veto', () => {
const row = use({ 'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', marker: 'plan', extra: 'Not this: `<!-- arbeidsform: ordre -->` in a sentence.' }) },
{ records: [briefRun({ order_id: 'a' })] });
assert.equal(row.facts.declaredNo, 0);
assert.equal(row.value, 1);
});
test('use: arbeidsform: ordre on an ikke-nytt or unmarked order adds nothing to the denominator', () => {
const row = use({
'r/orders/y.md': order({ id: 'y', date: '2026-09-29T00:00:00Z', marker: 'plan' }),
'r/orders/i.md': order({ id: 'i', date: '2026-09-29T00:00:00Z', marker: 'ikke-nytt', arbeidsform: 'ordre' }),
'r/orders/u.md': order({ id: 'u', date: '2026-09-29T00:00:00Z', arbeidsform: 'ordre' }),
}, { records: [briefRun({ order_id: 'y' })] });
assert.equal(row.facts.declaredNo, 0);
assert.equal(row.denominator, 1);
});
test('use: arbeidsform: voyage without a record is NOT MEASURED - the declaration is never the source', () => {
const row = use({ 'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', marker: 'plan', arbeidsform: 'voyage' }) });
assert.equal(row.facts.withVoyageRun, 0);
assert.equal(row.facts.noRecord, 1);
assert.equal(row.countable, false);
});
test('use: orders before the counting start are NOT MEASURED; an order dated exactly on it counts', () => {
const row = use({
'r/orders/early.md': order({ id: 'early', date: '2026-10-01T23:59:59Z', marker: 'plan' }),
'r/orders/on.md': order({ id: 'on', date: '2026-10-02T00:00:00Z', marker: 'plan' }),
}, { countingSince: '2026-10-02', records: [briefRun({ order_id: 'early', slug: 'e' }), briefRun({ order_id: 'on' })] });
assert.equal(row.facts.beforeMarker, 1);
assert.equal(row.value, 1);
assert.equal(row.denominator, 1);
assert.deepEqual(row.facts.matched, ['on']);
});
test('use: with no counting start the row is NOT COUNTABLE and every order is NOT MEASURED', () => {
const row = use({
'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', marker: 'plan' }),
'r/orders/b.md': order({ id: 'b', date: '2026-09-29T00:00:00Z', marker: 'plan', arbeidsform: 'ordre' }),
}, { countingSince: null, records: [briefRun({ order_id: 'a' })] });
assert.equal(row.countable, false);
assert.equal(row.value, null);
assert.equal(row.facts.notMeasured, row.facts.scanned);
assert.equal(row.facts.countingSince, null);
assert.match(row.detail, /counting has not started/);
assert.match(row.source, /counting not started/);
});
test('use: the window starts at the latest of the cutoff, MARKER_SINCE and the counting start', () => {
const files = { 'r/orders/a.md': order({ id: 'a', date: '2026-10-04T00:00:00Z', marker: 'plan' }) };
assert.match(use(files, { cutoff: '2026-10-05', countingSince: '2026-10-03' }).source, /from 2026-10-05/);
assert.match(use(files, { cutoff: '2026-09-21', countingSince: '2026-10-03' }).source, /from 2026-10-03/);
});
test('use: the exact outcome over YES, declared NO, no record and disputed', () => {
const row = use({
'r/orders/y.md': order({ id: 'y', date: '2026-09-29T00:00:00Z', marker: 'plan' }),
'r/orders/n.md': order({ id: 'n', date: '2026-09-29T00:00:00Z', marker: 'prove', arbeidsform: 'ordre' }),
'r/orders/r.md': order({ id: 'r', date: '2026-09-29T00:00:00Z', marker: 'plan', arbeidsform: 'voyage' }),
'r/orders/d.md': order({ id: 'd', date: '2026-09-29T00:00:00Z', marker: 'plan' }),
'r/orders/u.md': order({ id: 'u', date: '2026-09-29T00:00:00Z' }),
'r/orders/e.md': order({ id: 'e', date: '2026-09-01T00:00:00Z', marker: 'plan' }),
}, { records: [briefRun({ order_id: 'y', slug: 'y' }), briefRun({ order_id: 'd', slug: 'd1' }), briefRun({ order_id: 'd', slug: 'd2' })] });
assert.equal(row.countable, true);
assert.equal(row.value, 1);
assert.equal(row.denominator, 2, 'y (YES) + n (declared NO)');
assert.equal(row.facts.withVoyageRun, 1);
assert.equal(row.facts.declaredNo, 1);
assert.equal(row.facts.noRecord, 1);
assert.equal(row.facts.disputed, 1);
assert.equal(row.facts.notMeasured, 4, 'r (no record) + d (disputed) + u (unmarked) + e (before the window)');
assert.match(row.detail, /value = 1 of 2 measured/);
});
test('use: ordersNamedByARecord counts named orders even before counting starts (the known-positive check)', () => {
const row = use({
'r/orders/a.md': order({ id: 'a', date: '2026-09-29T00:00:00Z', marker: 'plan' }),
'r/orders/b.md': order({ id: 'b', date: '2026-09-29T00:00:00Z', marker: 'plan' }),
}, { countingSince: null, records: [briefRun({ order_id: 'a' })] });
assert.equal(row.countable, false);
assert.equal(row.facts.ordersNamedByARecord, 1);
});
// --- 3 · INTERRUPTS ----------------------------------------------------------
test('interrupts: no user_input record means NOT COUNTABLE, with the line denominator stated', () => {
const dir = fixture({
'trekexecute-stats.jsonl': jsonl({ ts: 't', session_id: 's', success: true }, { ts: 't', event: 'main-merge-gate', payload: {} }),
'trekplan-stats.jsonl': jsonl({ ts: 't', slug: 'x' }),
});
try {
const row = measureInterrupts({ stats: loadStats(dir), callSites: { sites: [], scanned: 3 } });
assert.equal(row.countable, false);
assert.equal(row.facts.lines, 3);
assert.equal(row.facts.userInput, 0);
assert.match(row.detail, /0 user_input records of 3 lines in 2 files/);
assert.match(row.detail, /0 emitter call sites/);
} finally { cleanup(dir); }
});
test('interrupts: user_input WITHOUT a slug or run id is still not countable per delivery', () => {
const dir = fixture({
'trekexecute-stats.jsonl': jsonl({ ts: 't', event: 'user_input', payload: { question: 'q' } }),
});
try {
const row = measureInterrupts({ stats: loadStats(dir), callSites: { sites: [], scanned: 1 } });
assert.equal(row.countable, false);
assert.equal(row.facts.userInput, 1);
assert.equal(row.facts.withDelivery, 0);
} finally { cleanup(dir); }
});
test('interrupts: user_input carrying a slug or run id is countable, per delivery', () => {
const dir = fixture({
'trekexecute-stats.jsonl': jsonl(
{ ts: 't1', event: 'user_input', payload: { slug: 'feat-a' } },
{ ts: 't2', event: 'user_input', payload: { slug: 'feat-a' } },
{ ts: 't3', event: 'user_input', payload: { run_id: 'r-9' } },
),
});
try {
const row = measureInterrupts({ stats: loadStats(dir), callSites: { sites: [], scanned: 1 } });
assert.equal(row.countable, true);
assert.equal(row.value, 3);
assert.deepEqual(row.facts.perDelivery, { 'feat-a': 2, 'r-9': 1 });
} finally { cleanup(dir); }
});
test('emitterCallSites: counts user_input emissions outside lib/stats/ and tests/ only', () => {
const root = fixture({
'commands/a.md': 'node ${CLAUDE_PLUGIN_ROOT}/lib/stats/event-emit.mjs \\\n --event user_input \\\n',
'commands/b.md': 'the emitter knows user_input as an event name - prose, not a call\n',
'lib/util/x.mjs': "emitEvent('user_input', { slug });\n",
'lib/stats/event-emit.mjs': "emit('user_input', {});\n",
'tests/lib/y.test.mjs': "emit('user_input', {});\n",
'hooks/scripts/h.mjs': "spawnSync('node', [EMIT, '--event', 'user_input']);\n",
});
try {
const r = emitterCallSites(root);
assert.deepEqual(r.sites.map((s) => s.file).sort(), ['commands/a.md', 'hooks/scripts/h.mjs', 'lib/util/x.mjs']);
assert.equal(r.scanned, 4, 'commands/a.md, commands/b.md, lib/util/x.mjs, hooks/scripts/h.mjs');
} finally { cleanup(root); }
});
// --- 4 · the signature-preserving mutant (M9) ----------------------------------
const MODULE = 'export function verdict(code) { return { status: code === 0 ? "PASSED" : "FAILED" }; }\n';
const STRONG_TEST = [
"import { test } from 'node:test';",
"import { strict as assert } from 'node:assert';",
"import { verdict } from '../lib/m.mjs';",
"test('verdict fells a failing exit', () => { assert.equal(verdict(1).status, 'FAILED'); });",
'',
].join('\n');
const behaviourRegistry = {
defects: [{
id: 'D-X', probe: 'behaviour',
check: [{ test: 'tests/t.test.mjs', name: 'verdict fells a failing exit', expect: 'fails', stubs: ['lib/m.mjs'] }],
}],
experiments: [],
};
test('buildM9Tree: every export keeps its name and returns {status:"PASSED"}; every named test only checks typeof', async () => {
const root = fixture({ 'lib/m.mjs': MODULE, 'tests/t.test.mjs': STRONG_TEST });
const m9 = buildM9Tree(root, behaviourRegistry);
try {
const mod = readFileSync(join(m9, 'lib', 'm.mjs'), 'utf8');
assert.match(mod, /export function verdict\(\) \{ return \{ status: 'PASSED' \}; \}/);
const t = readFileSync(join(m9, 'tests', 't.test.mjs'), 'utf8');
assert.match(t, /verdict fells a failing exit/);
assert.match(t, /typeof/);
assert.doesNotMatch(t, /FAILED/, 'the mutant test asserts nothing about behaviour');
assert.equal(readFileSync(join(root, 'lib', 'm.mjs'), 'utf8'), MODULE, 'the measured tree is never touched');
} finally { cleanup(root, m9); }
});
test('mutants: a gate that CLOSES the probe on M9 has felled 0 of 1', () => {
const root = fixture({ 'lib/m.mjs': MODULE, 'tests/t.test.mjs': STRONG_TEST });
try {
const row = measureMutants(root, {
registry: behaviourRegistry,
evaluate: () => ({ status: 'closed', detail: '' }),
runTest: () => true,
});
assert.equal(row.countable, true);
assert.equal(row.value, 0);
assert.equal(row.denominator, 1);
assert.match(row.detail, /D-X closed/);
} finally { cleanup(root); }
});
test('mutants: a gate that refuses to close the probe on M9 has felled 1 of 1', () => {
const root = fixture({ 'lib/m.mjs': MODULE, 'tests/t.test.mjs': STRONG_TEST });
try {
const row = measureMutants(root, {
registry: behaviourRegistry,
evaluate: () => ({ status: 'not-fellable', detail: 'still passes against a signature-preserving mutant' }),
runTest: () => true,
});
assert.equal(row.value, 1);
assert.equal(row.denominator, 1);
} finally { cleanup(root); }
});
test('mutants: a mutant whose own tests do not pass is not LIVE, so the row cannot be counted', () => {
// Without this control a broken fixture would read as "felled".
const root = fixture({ 'lib/m.mjs': MODULE, 'tests/t.test.mjs': STRONG_TEST });
try {
const row = measureMutants(root, {
registry: behaviourRegistry,
evaluate: () => ({ status: 'not-fellable', detail: 'x' }),
runTest: () => null,
});
assert.equal(row.countable, false);
assert.equal(row.value, null);
assert.match(row.detail, /not live/);
} finally { cleanup(root); }
});
test('mutants: against the REAL gate and registry, M9 is live and the denominator is 1', () => {
// Pins the measurement's shape, not today's value: M9's typeof-only tests
// really do pass on the M9 tree (so "felled" can only come from the gate).
const row = measureMutants(ROOT);
assert.equal(row.countable, true, row.detail);
assert.equal(row.denominator, 1);
assert.ok(row.value === 0 || row.value === 1);
assert.deepEqual(row.facts.entries.sort(), ['D-03', 'D-04']);
});
// --- the whole yardstick -------------------------------------------------------
function world({ orders = {}, stats = {} } = {}) {
const coord = fixture(Object.keys(orders).length ? orders : { 'r/orders/.keep': '' });
const data = fixture(Object.keys(stats).length ? stats : { '.keep': '' });
return { coord, data };
}
test('measureAll + main: RED with exit 1 while any number is not countable', () => {
const { coord, data } = world({
orders: { 'r/orders/o.md': order({ id: 'o', date: '2026-09-22T00:00:00Z' }) },
stats: { 'trekreview-stats.jsonl': jsonl(review('2026-09-01T00:00:00Z', 'ALLOW')) },
});
try {
const result = measureAll({ root: ROOT, coordDir: coord, dataDir: data, cutoff: DEFAULT_CUTOFF });
assert.equal(result.green, false);
assert.equal(result.countable, 1);
const text = render(result);
assert.match(text, /1 of 3 numbers countable/);
assert.match(text, /signature-preserving mutants felled: \d of 1/);
assert.match(text, /verdict: NOT JUDGED/);
for (const id of ['use', 'trust', 'interrupts', 'mutants']) assert.match(text, new RegExp(`\\| ${id} \\|`));
assert.equal(main(['--root', ROOT, '--coord', coord, '--data', data, '--json']), 1);
} finally { cleanup(coord, data); }
});
test('measureAll: all three countable and the mutant felled is still RED until the verdict can be judged', () => {
const { coord, data } = world({
orders: { 'r/orders/o.md': order({ id: 'o', date: '2026-09-29T00:00:00Z', marker: 'plan' }) },
stats: {
'trekbrief-stats.jsonl': jsonl(briefRun({ order_id: 'o', slug: 'a' })),
'trekreview-stats.jsonl': jsonl(review('2026-09-01T00:00:00Z', 'ALLOW', 'a')),
'trekexecute-stats.jsonl': jsonl({ ts: 't', event: 'user_input', payload: { slug: 'a' } }),
},
});
try {
const result = measureAll({
root: ROOT, coordDir: coord, dataDir: data, cutoff: DEFAULT_CUTOFF, countingSince: MARKER_SINCE,
mutantOptions: { evaluate: () => ({ status: 'not-fellable', detail: '' }) },
});
assert.equal(result.countable, 3);
assert.equal(result.rows.find((r) => r.id === 'mutants').value, 1);
assert.equal(result.verdict.judged, false);
assert.equal(result.green, false);
} finally { cleanup(coord, data); }
});
test('main: an unknown argument or a malformed cutoff is a usage error (exit 2)', () => {
assert.equal(main(['--nope']), 2);
assert.equal(main(['--cutoff', '21.09.2026']), 2);
});
test('measureAll: without countingSince it uses USE_COUNTING_SINCE, whatever its value', () => {
assert.ok('USE_COUNTING_SINCE' in Y, 'scripts/yardstick.mjs exports USE_COUNTING_SINCE');
const { coord, data } = world({
orders: { 'r/orders/o.md': order({ id: 'o', date: '2026-10-29T00:00:00Z', marker: 'plan' }) },
stats: { 'trekbrief-stats.jsonl': jsonl(briefRun({ order_id: 'o' })) },
});
try {
const all = measureAll({ root: ROOT, coordDir: coord, dataDir: data, cutoff: DEFAULT_CUTOFF,
mutantOptions: { evaluate: () => ({ status: 'not-fellable', detail: '' }) } });
const direct = measureUse({ queue: readOrders(coord), runs: Y.orderRuns(loadStats(data)), cutoff: DEFAULT_CUTOFF,
countingSince: Y.USE_COUNTING_SINCE });
assert.deepEqual(all.rows.find((r) => r.id === 'use'), direct);
} finally { cleanup(coord, data); }
});
test('main: the CLI prints its result when run through a symlink', () => {
const { coord, data } = world();
const links = mkdtempSync(join(tmpdir(), 'yardstick-link-'));
try {
const link = join(links, 'yardstick.mjs');
symlinkSync(join(ROOT, 'scripts', 'yardstick.mjs'), link);
const r = spawnSync(process.execPath, [link, '--coord', coord, '--data', data, '--json'], { encoding: 'utf8', timeout: 120000 });
assert.ok(r.stdout.trim().length > 0, `no output through a symlink (exit ${r.status})`);
assert.ok(Array.isArray(JSON.parse(r.stdout).rows));
} finally { cleanup(coord, data, links); }
});

51
tests/validators/intent-approval.test.mjs
View file

@ -8,7 +8,7 @@
// (a) brief without a marker → /trekplan's check exits non-zero
// (b) marker + intent text edited after → exits non-zero (stale)
// (c) valid marker → exits 0 (known negative)
// (d) the stamp runs before the manual/auto fork and emits brief-approved
// (d) the stamp runs before the manual/auto fork
//
// The module is imported dynamically so a missing module fails each test on its
// own instead of crashing the file into a single failure.
@ -86,12 +86,6 @@ function gate(briefPath) {
return run(commandLine('commands/trekplan.md', '--check'), { '{brief_path}': briefPath });
}
function stampInto(dir, env) {
// {BRIEF_PATH} since M2 (the resolved --approve target); {PROJECT_DIR} kept for the old spelling
return run(commandLine('commands/trekbrief.md', '--stamp'),
{ '{BRIEF_PATH}': join(dir, 'brief.md'), '{PROJECT_DIR}': dir }, env);
}
function codes(res) {
const out = JSON.parse(res.stdout);
return out.errors.map((e) => e.code);
@ -223,7 +217,7 @@ test('trekplan prose: the intent-approval gate sits in "Read the brief" and says
assert.match(section, /\*\*halt\*\*/i);
});
// ---- (d) the stamp: operator answer → marker + brief-approved, on BOTH paths ----
// ---- (d) the stamp: operator answer → marker, on BOTH paths ----
test('(d) trekbrief: the stamp step runs BEFORE the Phase 5 manual/auto fork', () => {
const text = read('commands/trekbrief.md');
@ -239,47 +233,6 @@ test('(d) trekbrief: the stamp step runs BEFORE the Phase 5 manual/auto fork', (
assert.match(before, /--quick/);
});
test('(d) trekbrief: the auto path no longer emits brief-approved on its own (one emission, not two)', () => {
const text = read('commands/trekbrief.md');
const auto = text.indexOf('### Auto path');
const phase6 = text.indexOf('## Phase 6');
assert.ok(auto > 0 && phase6 > auto);
assert.ok(!/--event brief-approved/.test(text.slice(auto, phase6)),
'brief-approved is emitted by the stamp; a second emission on the auto path would double-count');
});
test('(d) the manual-path stamp line writes the marker AND emits a brief-approved record', () => {
const dir = tmpDir();
const data = tmpDir();
try {
writeFileSync(join(dir, 'brief.md'), BRIEF);
const r = stampInto(dir, { CLAUDE_PLUGIN_DATA: data });
assert.equal(r.status, 0, `stamp failed: ${r.stdout} ${r.stderr}`);
const after = readFileSync(join(dir, 'brief.md'), 'utf8');
assert.match(after, /^intent_approved_hash: "sha256:[0-9a-f]{64}"$/m);
assert.match(after, /^intent_approved_at: "\d{4}-\d\d-\d\dT[^"]+"$/m);
const statsFile = join(data, 'trekexecute-stats.jsonl');
assert.ok(existsSync(statsFile), 'brief-approved record must be written');
const recs = readFileSync(statsFile, 'utf8').trim().split('\n').map((l) => JSON.parse(l));
const ev = recs.filter((x) => x.event === 'brief-approved');
assert.equal(ev.length, 1);
assert.equal(ev[0].payload.slug, 'dark-mode-toggle');
assert.match(ev[0].payload.intent_hash, /^sha256:[0-9a-f]{64}$/);
// and the gate now passes on the stamped file
assert.equal(gate(join(dir, 'brief.md')).status, 0);
} finally {
rmSync(dir, { recursive: true, force: true });
rmSync(data, { recursive: true, force: true });
}
});
test('(d) with CLAUDE_PLUGIN_DATA unset the record still lands (fallback = the dir the yardstick reads)', async () => {
const { resolveApprovalDataDir } = await load();
const d = resolveApprovalDataDir({ HOME: '/home/someone' });
assert.equal(d, '/home/someone/.claude/plugins/data/voyage-ktg-plugin-marketplace');
assert.equal(resolveApprovalDataDir({ CLAUDE_PLUGIN_DATA: '/x/y', HOME: '/h' }), '/x/y');
});
test('re-stamping replaces the marker instead of duplicating it', async () => {
const { stampIntentApproval } = await load();
const once = stampIntentApproval(BRIEF, new Date('2026-09-23T08:00:00Z')).text;