fix(route): measure the effort a session ran with, instead of asking for it
The last-session record exists to make the routing policy falsifiable, and it only is if --last-effort is measured. route.sh documented the opposite as settled fact: that the effort a session ran with is not observable from inside that session. That was true when written and is not now. Claude Code exports CLAUDE_EFFORT into every tool-use context as the session's current effort level, so a Bash call reads it directly. The premise had a cost. With effort unobservable, the record could only be completed by asking the operator at session end, which made it block on their presence -- all four fields or none. That is also the weaker measurement, and in the same way the previous board line is: the operator reads the effort off the startup command they typed, so both sources report what was PRESCRIBED rather than what was RUN. They come apart exactly when the record would be most interesting, which is what a session that silently ran xhigh under a board line saying high already showed. Reading it makes all four fields knowable from inside the ending session, so the record no longer waits on anyone. The skill does the reading; route.sh deliberately does NOT default from the variable, because a calculator that consults its environment is no longer deterministic from its arguments and the route->board round trip in selftest section 6 rests on that. Section 13 also pins the trap this opens: skill frontmatter overrides the session effort while that skill is active, so an effort: field in route's own SKILL.md would make the reading report the skill instead of the session -- a measurement quietly measuring itself, with nothing in the output to show it happened. Also corrects the neighbouring claim that the model is readable from the environment. There is no CLAUDE_MODEL; the session takes it from what it knows itself to be running as. route-selftest 50 -> 56. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gfa1nvwGXdST2MHvbs6htD
This commit is contained in:
parent
6735298752
commit
ac62a38b40
5 changed files with 96 additions and 17 deletions
16
CLAUDE.md
16
CLAUDE.md
|
|
@ -53,7 +53,7 @@ marketplace plugin. Three components, one boundary:
|
|||
rationale, and prints one block of `key=value` lines: the rubric row, the rule
|
||||
that fired, the `next-cost` value, a pasteable startup command, the one-row
|
||||
cheaper fallback, and the STATE.md comment lines. Pinned by
|
||||
`route-selftest.sh` (50 checks).
|
||||
`route-selftest.sh` (56 checks).
|
||||
|
||||
**It is here because it is the WRITER for the field `board.sh` already reads.**
|
||||
`next-cost` had a reader and no writer, so it was hand-typed every session and
|
||||
|
|
@ -70,6 +70,18 @@ marketplace plugin. Three components, one boundary:
|
|||
*grammar* and points here for the *values*. Scoring the traits is judgement
|
||||
and belongs to the skill; turning scores into a row is a lookup and takes zero
|
||||
model calls.
|
||||
|
||||
**`--last-effort` is MEASURED from `CLAUDE_EFFORT`, and the calculator must
|
||||
never default it.** Claude Code exports that variable into every tool-use
|
||||
context as the session's current effort, so the caller reads it and passes it
|
||||
in; having `route.sh` read it directly would make the output depend on the
|
||||
environment instead of on its arguments, and the round trip in selftest
|
||||
section 6 rests on that determinism. The two sources it replaces fail
|
||||
identically: the previous board line holds what was *prescribed*, and asking
|
||||
the operator launders that same prescription through someone reading their own
|
||||
startup command. Corollary pinned by section 13: `skills/route/SKILL.md` must
|
||||
never declare an `effort:` frontmatter field, because frontmatter overrides the
|
||||
session effort and the reading would then measure the skill, not the session.
|
||||
- **Skills (`skills/coord-send/`, `skills/board/`, `skills/route/`):** natural-language front
|
||||
doors mapping user intent to engine invocations. No mailbox logic lives here
|
||||
either. `board` additionally owns the *ranking* — which repo wins and why —
|
||||
|
|
@ -119,7 +131,7 @@ obligations in another repo.
|
|||
- TDD: no behavior change without a failing selftest check first.
|
||||
`bash scripts/coord-selftest.sh` must exit 0 (156/156),
|
||||
`bash scripts/board-selftest.sh` must exit 0 (36/36) and
|
||||
`bash scripts/route-selftest.sh` must exit 0 (50/50).
|
||||
`bash scripts/route-selftest.sh` must exit 0 (56/56).
|
||||
- English for all code, docs, and commit messages (public repo). Norwegian
|
||||
trigger aliases in the skill description are deliberate.
|
||||
- Conventional Commits: `type(scope): description`.
|
||||
|
|
|
|||
|
|
@ -134,7 +134,7 @@ Note that raising the inbox's priority (Rule 7) deliberately does **not** widen
|
|||
|
||||
bash scripts/coord-selftest.sh # 156 checks against a throwaway mailbox
|
||||
bash scripts/board-selftest.sh # 36 checks against a throwaway repo tree
|
||||
bash scripts/route-selftest.sh # 50 checks, incl. the route->board round trip
|
||||
bash scripts/route-selftest.sh # 56 checks, incl. the route->board round trip
|
||||
npm test # all three selftests via node --test
|
||||
|
||||
TDD is the house rule: every behavior change lands with a failing selftest check first.
|
||||
|
|
|
|||
|
|
@ -371,6 +371,44 @@ check "--help shows no versionless model example" "$rc"
|
|||
if printf '%s' "$HELP" | grep -q 'blocked-on='; then rc=1; else rc=0; fi
|
||||
check "--help does not restate the board line grammar" "$rc"
|
||||
|
||||
# --- 13. Where --last-effort comes from -----------------------------------
|
||||
# The record exists to make the policy falsifiable, which it only is if the
|
||||
# effort field is MEASURED. Two earlier sources both measured the wrong thing:
|
||||
# the previous board line holds what was PRESCRIBED, and asking the operator
|
||||
# launders that same prescription through a human who is reading it off the
|
||||
# startup command they typed. $CLAUDE_EFFORT is what the session actually
|
||||
# resolved - it is documented as the CURRENT effort level and is exported into
|
||||
# every tool-use context, which is why a Bash call can read it at all.
|
||||
#
|
||||
# The trap this section exists to pin: skill frontmatter can set `effort`, and
|
||||
# frontmatter overrides the session level while that skill is active. An
|
||||
# `effort:` field in route's own SKILL.md would therefore make the reading
|
||||
# report the SKILL's effort and not the session's - a measurement quietly
|
||||
# measuring itself, with nothing in the output to show it happened.
|
||||
SKILL="$DIR/../skills/route/SKILL.md"
|
||||
[ -f "$SKILL" ]; check "route SKILL.md is where the selftest expects it" $?
|
||||
|
||||
# Frontmatter only: the body must be free to DISCUSS effort at length.
|
||||
FM="$(awk 'NR==1 && /^---$/ {f=1; next} f && /^---$/ {exit} f {print}' "$SKILL" 2>/dev/null)"
|
||||
if printf '%s' "$FM" | grep -q '^effort:'; then rc=1; else rc=0; fi
|
||||
check "route SKILL.md declares no effort: frontmatter field" "$rc"
|
||||
|
||||
grep -q 'CLAUDE_EFFORT' "$SKILL"
|
||||
check "route SKILL.md names CLAUDE_EFFORT as the --last-effort source" $?
|
||||
|
||||
grep -q 'board line' "$SKILL"
|
||||
check "route SKILL.md still warns off the previous board line" $?
|
||||
|
||||
# route.sh carried the claim as a documented premise. It was true when written
|
||||
# and is not any more, so it must not survive as a comment that reads like a
|
||||
# measured fact three weeks from now.
|
||||
FLAT="$(tr '\n' ' ' < "$R" | sed 's/#//g' | tr -s ' ')"
|
||||
if printf '%s' "$FLAT" | grep -q 'is not observable from inside'; then rc=1; else rc=0; fi
|
||||
check "route.sh no longer claims effort is unobservable from inside" "$rc"
|
||||
|
||||
grep -q 'CLAUDE_EFFORT' "$R"
|
||||
check "route.sh names the source the caller should measure from" $?
|
||||
|
||||
echo ""
|
||||
echo "route-selftest: $PASS passed, $FAIL failed"
|
||||
[ "$FAIL" -eq 0 ] || exit 1
|
||||
|
|
|
|||
|
|
@ -91,11 +91,16 @@
|
|||
# zero on row 4 means escalation fires too readily.
|
||||
#
|
||||
# All four fields are closed sets or numbers, and required together, because the
|
||||
# NEXT session reads the record back to decide --opus-xhigh-failed. Note what
|
||||
# the caller must supply rather than infer: the effort a session ran with is not
|
||||
# observable from inside that session, and reading it back off the previous
|
||||
# board line would measure what was PRESCRIBED, not what was RUN. Omit the
|
||||
# record rather than guess - a guessed value reads back as a measurement.
|
||||
# NEXT session reads the record back to decide --opus-xhigh-failed. Every one of
|
||||
# them must be MEASURED by the caller: --last-effort comes from CLAUDE_EFFORT,
|
||||
# which Claude Code exports into every tool-use context as the session's current
|
||||
# effort level. It is deliberately NOT defaulted from that variable here - a
|
||||
# calculator that reads its own environment stops being deterministic from its
|
||||
# arguments, and section 6 of the selftest depends on that determinism. Reading
|
||||
# the value off the previous board line would measure what was PRESCRIBED rather
|
||||
# than what was RUN; so, less obviously, does asking the operator, who reads it
|
||||
# off the startup command they typed. Omit the record rather than guess - a
|
||||
# guessed value reads back as a measurement.
|
||||
#
|
||||
# It is deliberately NOT wired to the Fable rows. Inferring "the model failed"
|
||||
# from "the session did not finish" would fire on context exhaustion and on
|
||||
|
|
|
|||
|
|
@ -109,15 +109,39 @@ in the session that is ending — never from what STATE.md prescribed:
|
|||
- `--last-corrections <n>` — how many rounds of rework it took. This is the
|
||||
cheap proxy for whether the routing was right.
|
||||
|
||||
**Effort is not observable from inside a running session.** The model is
|
||||
readable from the environment; the effort level the operator launched with is
|
||||
not. So do not infer it, and in particular do not read it back from the previous
|
||||
board line — that measures what was *prescribed*, not what was *run*, and the
|
||||
two come apart exactly when the record would be most interesting. Ask the
|
||||
operator for the effort level when closing the session. If they are not there to
|
||||
ask, **omit the record entirely** — it is all four fields or none. A guessed
|
||||
effort is worse than a missing one, because it reads back later as a
|
||||
measurement.
|
||||
**Measure the effort, never infer it.** Read what this session actually
|
||||
resolved:
|
||||
|
||||
```bash
|
||||
echo "$CLAUDE_EFFORT"
|
||||
```
|
||||
|
||||
`CLAUDE_EFFORT` is Claude Code's own *current* effort level, exported into every
|
||||
tool-use context — which is why a Bash call can read it. Pass it verbatim as
|
||||
`--last-effort`. The model is not in the environment (there is no
|
||||
`CLAUDE_MODEL`); take it from what this session knows itself to be running as.
|
||||
|
||||
Two sources are **wrong on purpose**, and both fail the same way. The previous
|
||||
board line holds what was *prescribed*, not what was *run* — the two come apart
|
||||
exactly when the record would be most interesting. Asking the operator launders
|
||||
that same prescription through a human, who is reading it off the startup
|
||||
command they typed rather than off the running process. Confirming a measured
|
||||
value with them is fine; sourcing it from them is not.
|
||||
|
||||
So the record no longer waits on anyone: all four fields are knowable from
|
||||
inside the session that is ending. Still omit it entirely — all four or none —
|
||||
if any one of them is genuinely unknown. A guessed value is worse than a missing
|
||||
one, because it reads back later as a measurement.
|
||||
|
||||
**This file must never declare an `effort:` frontmatter field.** Skill
|
||||
frontmatter overrides the session effort while the skill is active, so the
|
||||
reading above would report *this skill's* effort instead of the session's — a
|
||||
measurement measuring itself, with nothing in the output to show it happened.
|
||||
Pinned by `route-selftest.sh` section 13.
|
||||
|
||||
One honest limit: `CLAUDE_EFFORT` is the *current* level, so if the operator
|
||||
changed it mid-session with `/effort`, "the effort this session ran with" is not
|
||||
a single value. Record the level the work was actually done at and say so.
|
||||
|
||||
Read the previous `route-last` line out of STATE.md before overwriting it.
|
||||
Pass `--opus-xhigh-failed` **only** when it says an `Opus 5`/`xhigh` session ran
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue