Kjell Tore Guttormsen
e2aec019ac
computeVerdict counted only the findings handed to it (reasoned.kept), so a
finding removed by Pass 2 or Pass 3, and a reviewer whose payload was thrown
away or never arrived, were arithmetically identical to a finding that never
existed. All three pushed the verdict toward ALLOW.
Measured before the fix (probes, 2026-09-01):
- a BLOCKER with a 101-character title -> ALLOW (Pass 2 succinctness)
- a payload with one ad-hoc rule_key is skipped WHOLE at ingest, taking a
valid BLOCKER sibling with it -> ALLOW
- a reviewer that never reported -> ALLOW
Pass 3's own no-citation / unknown-rule_key branches turned out unreachable
through runContract (validateFindings rejects those payloads first), so the
reachable exposure was Pass 2 plus the skipped/absent reviewer.
THE OPEN DESIGN DECISION, and why it went against the order's default.
The order proposed: indeterminate file-existence YES, plain succinctness NO
("a too-long finding is not an uncertain finding"). I kept the first and
overrode the second, on one principle:
A removal is `dropped` only when the test REFUTED the finding as a claim
about this codebase. Every other removal is `unverified`.
Succinctness and actionability read a `.length`. They never examine the claim,
so they cannot establish the finding is unreal - and dropping a BLOCKER for a
101-character title is precisely the fail-open shape being fixed. Three things
settled it:
1. Under the order's default the fix would have been almost inert. Pass 3's
drop branches are unreachable via runContract, so leaving Pass 2 out would
have left the only reachable finding-level exposure open.
2. Cost asymmetry, priced rather than asserted: the verdict is not a gate.
Handover 6 feeds `findings` filtered to BLOCKER+MAJOR into /trekplan
(commands/trekplan.md:218); `verdict` is optional metadata
(docs/HANDOVER-CONTRACTS.md:353). Nothing loops or re-plans on WARN. So a
false `unverified` costs WARN plus a printed reason; a false drop costs a
silent ALLOW over a live BLOCKER.
3. unknown-rule_key joins them for the same reason: an ad-hoc key is a real
defect wearing the wrong label, and v5.1.1 high-effort mode already KEEPS
those, normalised to PLAN_EXECUTE_DRIFT. Refuting them at normal effort
while keeping them at high effort would be incoherent.
no-citation stays a drop: a finding whose file is empty or whose line is
negative names no location, so it makes no checkable claim at all - the one
deterministic refutation, and what the Pass 3 prose already said it was.
Iron Law: tests/lib/coordinator-contract.test.mjs first, red (missing export +
the three measured ALLOWs), then production code. Two existing assertions were
updated AFTER implementation as contract changes, not to make the red pass.
A known-positive control pins that ALLOW is still reachable - without it,
"no ALLOW" is not a fail-closed contract, only a broken one.
lib/review/coordinator-contract.mjs
+ classifySuppression / REFUTING_REASONS / UNVERIFIED_REASONS - one
vocabulary owned by the lib, including the tokens only the LLM
coordinator emits (accuracy:refuted, file-existence:refuted/indeterminate),
so prose and lib cannot drift. Unclassified reasons default to unverified:
the default fails closed.
~ judgeFilter / reasonablenessFilter return {kept, dropped, unverified}
~ computeVerdict(findings, {unverified, missingReviewers}) -> + allow_blocked_by.
Never raises a verdict, only withholds ALLOW. Unverified findings are NOT
counted into a severity tier: their severity was never substantiated, and
counting it would be invention.
~ runContract(payloads, {expectedReviewers}) -> + unverified,
missing_reviewers, allow_blocked_by. `suppressed` stays the union of
dropped + unverified, so existing consumers (gold-eval) keep their meaning.
agents/review-coordinator.md - Pass 2/3 tables gain a fate column, new
"Suppression is two-valued" section, Pass 4 threshold table gains the two
fail-closed rows, Executive Summary must state a withheld ALLOW, Suppressed
Findings tags each line [dropped]/[unverified]. Pass 3's unknown-rule_key
bullet explicitly says high-effort does not reach that branch, so the same
input never has two documented fates.
commands/trekreview.md - Phase 5 "Reviewer accounting": the expected set is
written down before the spawn, a silent reviewer gets one re-ask and then
STOP. That extends the pattern already in the file (schema failure -> 2
bounded re-asks -> "do not feed unvalidated findings to the coordinator") to
the other two ways a reviewer goes missing, rather than softening it to WARN.
The lib's missing_reviewers stays as belt-and-braces for direct callers.
docs/agent-return-channel-defect.md - the "inferred, not observed" caveat on
the unnamed arm above 66 lines is struck: akashic-intelligence S27
(f168630) measured 2/2 unnamed agents returning against a 4370-line plan,
30449 B and 10989 B, both valid JSON. Recorded with akashic's own two
caveats intact - the measurer owns the finding, and byte-identity between
the returned string and the file on disk was not proven. The separate S25
named-arm figures are left standing; these are two measurements, not a
correction of one by the other.
No release, no version bump, no tag, no catalogue ref, no Workflow port.
Suite 1025 (1023/0/2) -> 1034 (1032/0/2), 0 failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
268 lines
14 KiB
Markdown
268 lines
14 KiB
Markdown
# The Agent tool's `name` parameter silently destroys the return channel
|
||
|
||
**Date:** 2026-08-17 (voyage S85)
|
||
**Claude Code:** 2.1.226 · **Plugin:** voyage 5.9.1 (installed cache — verified byte-identical to the repo working tree for both reviewer agents before measuring)
|
||
**Status:** Root cause MEASURED and mechanistically confirmed. The defect is in
|
||
how the harness treats one Agent-tool parameter, not in voyage's agent
|
||
definitions. A one-line workaround exists and is now pinned in every voyage
|
||
spawn site.
|
||
|
||
## The report that triggered this
|
||
|
||
`akashic-intelligence` lost `/trekplan` Phase 9 twice. S25 (2026-08-17, plan
|
||
rev 5.0): four subagents spawned via the Agent tool — `voyage:plan-critic` ×2,
|
||
`voyage:scope-guardian` ×2, all `model: "opus"`, not Workflow. Nine
|
||
`SendMessage` reminders, zero replies. The round before had the same outcome
|
||
with tighter scope and an explicit tool-call budget. Consequence: the
|
||
hand-dedup step fell away (one source instead of two) and review coverage came
|
||
out **narrower** than the four preceding rounds.
|
||
|
||
Their own conclusion was "the channel, not the agents" — directionally right.
|
||
Two details in it are wrong and matter: they attributed it to the agents having
|
||
no `Write` tool, and they concluded the remedy is to abandon agents and run the
|
||
review inline. `Write` is irrelevant, and the agents do not need abandoning.
|
||
|
||
## Root cause
|
||
|
||
**Passing `name` to the Agent tool does not name a subagent. It changes what
|
||
kind of thing gets spawned.**
|
||
|
||
The spawn metadata says it outright. From
|
||
`~/.claude/projects/<project>/<session>/subagents/agent-*.meta.json`, same
|
||
session, same model, same `subagent_type`:
|
||
|
||
| spawn | recorded `agentType` | `taskKind` | `spawnDepth` |
|
||
|---|---|---|---|
|
||
| without `name` | `voyage:plan-critic` | *(none — a real subagent)* | `1` |
|
||
| with `name` | `pc-opus` (the name) | `in_process_teammate` | `0` |
|
||
|
||
The two kinds have different return semantics:
|
||
|
||
- **Subagent** (`spawnDepth: 1`) — the agent's final assistant text **is** the
|
||
return value. It arrives at the parent as a task notification.
|
||
- **Teammate** (`in_process_teammate`, `spawnDepth: 0`) — a peer of the main
|
||
session, not a child of it. Its plain final text is **not** a return value; it
|
||
is only transcript. A teammate reaches the parent **only** by calling
|
||
`SendMessage(to: "main")`.
|
||
|
||
Every voyage agent declares a `tools:` allowlist. `plan-critic` and
|
||
`scope-guardian` declare `["Read", "Glob", "Grep"]`. **No SendMessage.** As
|
||
teammates they are therefore *structurally* incapable of returning anything —
|
||
no prompt, no scope tightening, and no tool-call budget can change that. This
|
||
is exactly why akashic's second attempt with a stricter prompt failed
|
||
identically: the prompt was never the variable.
|
||
|
||
### Why it looks like a hung agent
|
||
|
||
The teammate runs. It reads the plan, reasons, and writes a complete, correct
|
||
final answer to its transcript. Then it stops. From the orchestrator's side
|
||
this is indistinguishable from an agent that stalled — so the natural response
|
||
is to poke it with `SendMessage`, which produces more transcript that also
|
||
never comes back. Nine reminders, zero replies.
|
||
|
||
### The symptom signature, exactly
|
||
|
||
The teammates do eventually surface — as content-free idle notifications,
|
||
delivered long after the fact. Observed in this session, all six named agents
|
||
(times UTC):
|
||
|
||
```
|
||
{"type":"idle_notification","from":"ctrl-pong", "idleReason":"available"} 19:57:30
|
||
{"type":"idle_notification","from":"ctrl-read", "idleReason":"available"} 19:57:35
|
||
{"type":"idle_notification","from":"pc-nooverride", "idleReason":"available"} 19:59:19
|
||
{"type":"idle_notification","from":"pc-opus", "idleReason":"available"} 19:59:24
|
||
{"type":"idle_notification","from":"sg-opus", "idleReason":"available"} 19:59:37
|
||
{"type":"idle_notification","from":"ctrl-sendback", "idleReason":"available",
|
||
"summary":"[to main] PONG-VIA-SENDMESSAGE"} 20:06:16
|
||
```
|
||
|
||
This is akashic's "innholdsløse idle-pings", verbatim, and it is the field
|
||
signature to recognise the defect by. Two things to read off it:
|
||
|
||
- **`summary` is present only for the teammate that called SendMessage.** For
|
||
the five tool-less ones the notification carries no content at all — not a
|
||
truncated result, not an error, nothing. The presence or absence of `summary`
|
||
is the fastest way to tell a delivering teammate from a mute one.
|
||
- **The idle notifications lag by minutes.** `ctrl-pong` went idle 10 seconds
|
||
after spawn; its notification arrived ~9 minutes later. So "no idle ping yet"
|
||
is not evidence the agent is still working, and the ping, when it comes, is
|
||
not a result. Judging liveness from these is how a completed run gets
|
||
re-prompted nine times.
|
||
|
||
**The work is not lost.** It is on disk at
|
||
`~/.claude/projects/<project-slug>/<session-id>/subagents/agent-a<name>-<hash>.jsonl`.
|
||
The final assistant text block in that file is the answer the orchestrator never
|
||
received. This is the recovery path when a run has already been burned.
|
||
|
||
## Measurements
|
||
|
||
One session, one target unless noted. Target for the reviewer cells:
|
||
`docs/devils-advocate-plan.md` (66 lines) — a real plan file, small enough that
|
||
runtime cannot be confused with hanging.
|
||
|
||
| # | `subagent_type` | `model` | `name`? | Returned to parent? | Time |
|
||
|---|---|---|---|---|---|
|
||
| 1 | `general-purpose` ("reply PONG", 0 tools) | `opus` | yes | **no** | >17 min, never |
|
||
| 2 | `general-purpose` (read file, count headings) | `opus` | yes | **no** | never |
|
||
| 3 | `voyage:plan-critic` | `opus` | yes | **no** | never |
|
||
| 4 | `voyage:scope-guardian` | `opus` | yes | **no** | never |
|
||
| 5 | `voyage:plan-critic` | *(no override)* | yes | **no** | never |
|
||
| 6 | `general-purpose`, told to call `SendMessage(to: "main")` | `opus` | yes | **yes** | ~30 s |
|
||
| 7 | `general-purpose` ("reply PONG", 0 tools) | `opus` | **no** | **yes** | **2.19 s** |
|
||
| 8 | `voyage:plan-critic` | `opus` | **no** | **yes** — full findings + JSON block | **110.3 s**, 7 tool uses |
|
||
| 9 | `voyage:scope-guardian` | `opus` | **no** | **yes** — full findings + JSON block | **109.1 s**, 8 tool uses |
|
||
|
||
Denominators, per varied dimension:
|
||
|
||
- **`name` present, no explicit SendMessage: 0/5 returned.** `name` absent:
|
||
**3/3 returned.** This is the only dimension that moved the outcome.
|
||
- **`name` present *with* an explicit `SendMessage(to: "main")` instruction:
|
||
1/1 returned** (cell 6). This is the positive control for the teammate
|
||
channel: it is not broken, it is merely never used by tool-less agents.
|
||
- **Model override: 2/2 non-returning.** Cells 3 (`model: "opus"`) and 5 (no
|
||
override) are identical in outcome — the override is not the variable, which
|
||
rules out akashic's stated `model: "opus"` detail as causal.
|
||
- **Agent type: 2/2 in each arm.** `plan-critic` and `scope-guardian` behave
|
||
identically named (both silent) and unnamed (both complete). The agent
|
||
definitions are not the variable.
|
||
- **Work actually performed while named: 5/5.** All five named agents produced
|
||
correct, complete final text in their transcripts within ~3 minutes —
|
||
`PONG`, `8`, plan-critic's blocker list with the correct legacy-format
|
||
advisory, and scope-guardian's coverage table. Nothing failed except delivery.
|
||
|
||
### Known-positive discipline (Verifiseringsloven face 4)
|
||
|
||
Two negative results in this investigation were produced by broken queries and
|
||
would have been consumed as facts:
|
||
|
||
1. **`ListAgents` returned no in-process subagents** while five were running. It
|
||
listed 26 peer sessions and none of mine. Probing one directly with
|
||
`SendMessage` proved the agents existed and had live inboxes. Had the empty
|
||
listing been believed, the diagnosis would have been "the spawns silently
|
||
no-op" — wrong.
|
||
2. **The first known-positive control failed.** Cell 1 (named PONG) was meant to
|
||
prove the spawn mechanism works, and it did not return either. A failed
|
||
control proves nothing; it only means the control shared the defect. The
|
||
control that discriminates is cell 7 — same prompt, same model, same agent
|
||
type, `name` removed — which returned in 2.19 s.
|
||
|
||
**Unmeasured cells, stated as unmeasured:** the returning arm was only exercised
|
||
against the 66-line target. Large targets (voyage's 573-line plan;
|
||
akashic's 3730-line / 277 KB `features/01-sun-position/plan.md`) were **not**
|
||
measured in either arm. Nothing here rules out a *separate*, size-dependent
|
||
failure at akashic's scale — it only establishes that the failure they observed
|
||
reproduces at 66 lines, where size cannot be the cause.
|
||
|
||
**External measurement closes half of that (2026-08-17).** `akashic-intelligence`
|
||
recovered its own S25 transcripts from
|
||
`~/.claude/projects/<slug>/<session>/subagents/` and read the last assistant
|
||
text block out of each. Denominator: 4 of 4 agents, all **named**, all against
|
||
the full 3730-line / 277 KB plan. Three produced valid JSON review output
|
||
(17, 8 and 13 findings); the fourth was their PONG control and produced
|
||
`PONG`. Their commit `e281a9d`.
|
||
|
||
What that closes: **the work is performed at 3730 lines.** No size-dependent
|
||
failure of the agent's *reasoning or output* exists at that scale — the named
|
||
arm produces correct final text at 66 lines and at 3730 lines alike, and only
|
||
delivery fails, identically at both. The 38 recovered findings were re-used
|
||
instead of re-run.
|
||
|
||
**A second external measurement closes the other half (akashic-intelligence
|
||
S27, commit `f168630`).** The gap left above was that all four S25 cells were
|
||
named, so the *returning* (unnamed) arm had no measurement above 66 lines.
|
||
S27 supplies one: denominator **2 of 2 unnamed agents**, against a 4370-line
|
||
plan; both returned, 30449 B and 10989 B, both valid JSON. The unnamed arm
|
||
therefore returns at full scale as observed fact, not as inference.
|
||
|
||
Two caveats, kept at the strength akashic itself stated them. The measurement
|
||
was taken by the repo that owns the finding, not by an independent third party.
|
||
And byte-identity between the returned string and the file on disk was not
|
||
proven — what is established is that a well-formed result of that size arrived,
|
||
not that it arrived unaltered.
|
||
|
||
Their PONG control also carries the same lesson as cell 1 above, in a third
|
||
repo: S25 reported that agent as having "gone idle without sending PONG". It
|
||
sent PONG. Absence of *delivery* was read as absence of *work* — the same
|
||
face-4 error, three times in one evening.
|
||
|
||
## Where the defect lives
|
||
|
||
**Outside this repository.** Nothing in `agents/plan-critic.md` or
|
||
`agents/scope-guardian.md` is wrong: correct frontmatter, a `tools:` list
|
||
appropriate to a read-only reviewer, a terminating prompt, and an output format
|
||
that both agents produced verbatim when spawned as real subagents. The failure
|
||
is a harness behaviour — a parameter that silently changes a spawn's kind and
|
||
its return contract, with no error, no warning, and no observable difference at
|
||
the call site beyond the wording of the tool result.
|
||
|
||
Per the standing rule, this is **not** worked around silently. It is documented
|
||
here, reported to `.claude` (which owns cross-repo quality and can escalate),
|
||
and guarded in voyage's own spawn instructions so no consumer re-enters it.
|
||
|
||
## In-repo fix
|
||
|
||
Every voyage command that spawns agents — `trekbrief`, `trekplan`,
|
||
`trekresearch`, `trekreview` — now states the rule at its spawn site, naming
|
||
the mechanism (`in_process_teammate`) so it cannot be mistaken for style
|
||
preference. `trekexecute` spawns nothing (Hard Rule 10) and is excluded.
|
||
|
||
Pinned by `tests/lib/doc-consistency.test.mjs`:
|
||
|
||
- `S85: the set of agent-spawning commands is exactly the four that carry the
|
||
no-name rule` — derives the spawning set from the command files themselves, so
|
||
the pin cannot go vacuous when a command starts or stops spawning.
|
||
- `S85: commands/<cmd>.md forbids the Agent tool's name parameter at its spawn
|
||
sites` — one per command.
|
||
|
||
Verified red before the prose was written, green after.
|
||
|
||
## Recommended working shape for consumers
|
||
|
||
For anyone running voyage's reviewers (akashic included), grounded in the
|
||
measurements above rather than in preference:
|
||
|
||
**1. Use agents. Spawn them without `name`.** Cells 8 and 9 are the evidence:
|
||
both reviewers returned complete, schema-conformant output — human-readable
|
||
findings *and* the machine-readable JSON block the dedup helper consumes — in
|
||
~110 seconds, in parallel, from a single message. This is the intended shape and
|
||
it works. Removing `name` is the entire fix.
|
||
|
||
**2. Do not switch to inline review as a remedy.** akashic's fallback was
|
||
rational under their diagnosis but it pays a real, measured price: with one
|
||
source there is no dedup step, and their own record shows `scope-guardian`
|
||
finding blockers in the rev 3.0/4.0 rounds that `plan-critic` did not see at
|
||
all. The two agents are not redundant — cells 8 and 9 here returned largely
|
||
*different* findings on the same 66-line plan (plan-critic led on
|
||
placeholder/headless defects; scope-guardian led on the plan already having been
|
||
executed and its deliverable never being written). Two independent passes with
|
||
different mandates is the property worth keeping, and unnamed agents deliver it
|
||
for free. Running two inline passes is the fallback if agents fail *for a
|
||
different, verified reason* — not for this one.
|
||
|
||
**3. If you deliberately want a named teammate, give it SendMessage and say so.**
|
||
Cell 6 shows the teammate channel works when used. That means adding
|
||
`SendMessage` to the agent's `tools:` list *and* instructing it in the prompt to
|
||
call `SendMessage(to: "main")` with its result. voyage does not do this: its
|
||
reviewers are read-only by design, and the subagent path already returns.
|
||
|
||
**4. A hard tool-call budget is not the lever here.** akashic tried a tighter
|
||
budget and it changed nothing, which this measurement explains: a 0-tool-call
|
||
agent (cell 1) failed exactly like a 7-tool-call one. Budget caps are a cost
|
||
control, not a liveness control.
|
||
|
||
**5. When an agent looks hung, check its transcript before re-prompting.** The
|
||
final assistant block in
|
||
`~/.claude/projects/<project-slug>/<session-id>/subagents/agent-*.jsonl` tells
|
||
you within seconds whether the agent failed or merely could not deliver. Reading
|
||
that file directly recovers a burned run's output. Extract only the last
|
||
assistant text block — these transcripts run to 140 KB and reading one whole
|
||
will flood the orchestrator's context.
|
||
|
||
## Open
|
||
|
||
- The `--gates`-adjacent "idle-agent gap" already logged as open operator
|
||
decision #3 in `STATE.md` (`/trekreview` Phase 5, `/trekplan` Phases 5/6/9,
|
||
`/trekresearch` Phase 4 have no empty-return detection) now has a measured
|
||
root cause for its most likely trigger. Whether to add active detection — as
|
||
opposed to the prevention pinned here — remains open and is not decided by
|
||
this document.
|