feat(route): give the advisor a writer, on a need and per row
route.sh now emits `--advisor opus` into the startup command it prints.
The flag existed and worked, but nothing generated it, so it went unused:
the only mechanism that ever set an advisor here was `/advisor`, which
writes the global advisorModel setting -- every session, every repo -- and
was abandoned for burning quota. Nothing replaced it.
Two independent triggers, almost disjoint by construction:
rows 1-2 always. Sonnet main model, so opus is a capability LIFT rather
than a peer. Load-bearing: every fallback is one row cheaper and
the cheap rows are Sonnet, so this makes the quota fallback safe.
rows 3-4 only at reversibility=costly|one-way. Opus main model, so the
advisor buys peer review where a mistake is not cheap to undo.
rows 5-6 never. The CLI rejects every advisor for a Fable main model.
costly forces row 3 and one-way forces row 4, so a Sonnet row always has
reversibility=cheap and neither rule reaches the other's rows.
verification=none is deliberately not a third trigger: beyond the stakes
rule it adds only cheap-to-reverse mistakes, docs sessions among them.
Applied per ROW, so fallback-command carries its own correct answer.
route-selftest.sh 56 -> 73. Section 14 gates the three CLI facts the rule
rests on against the installed claude without spending a token: advisor
validation runs before the empty-prompt check, so `-p ""` reaches the
validator and stops. --help cannot gate this -- it short-circuits before
option validation, so an unknown flag would pass the gate untested.
Three pre-existing checks updated rather than worked around: two asserted
whole command strings that now carry the advisor, and section 11's effort
extraction swallowed the tail of the command line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X8N8hQEJSWWtieWUx37txT
This commit is contained in:
parent
6ffe089b68
commit
9cb405c2cd
10 changed files with 272 additions and 13 deletions
|
|
@ -306,11 +306,13 @@ check "board + route + route-last stacked still yield prose in NESTE" $?
|
|||
# --- 9. Startup command and fallback --------------------------------------
|
||||
# Two spellings of ONE decision: the rubric name for the board line, the CLI
|
||||
# alias for the command the operator pastes. They must never disagree.
|
||||
# These assert the WHOLE string, advisor included, so section 14's rule cannot
|
||||
# be widened without a deliberate edit here - the exact-match is the tripwire.
|
||||
cmd="$(field partial strong cheap local command)"
|
||||
[ "$cmd" = "claude --model opus --effort high" ]; check "command mirrors the row (opus/high)" $?
|
||||
|
||||
cmd="$(field known strong cheap local command)"
|
||||
[ "$cmd" = "claude --model sonnet --effort high" ]; check "command mirrors the row (sonnet/high)" $?
|
||||
[ "$cmd" = "claude --model sonnet --effort high --advisor opus" ]; check "command mirrors the row (sonnet/high)" $?
|
||||
|
||||
# The rubric requires ALWAYS naming one row cheaper as the quota fallback.
|
||||
fb="$(field partial strong cheap local fallback)"
|
||||
|
|
@ -322,7 +324,7 @@ fb="$(field known strong cheap local fallback)"
|
|||
# The fallback needs its own pasteable command or the operator translates by
|
||||
# hand at exactly the moment they are under quota pressure.
|
||||
fbc="$(field partial strong cheap local fallback-command)"
|
||||
[ "$fbc" = "claude --model sonnet --effort xhigh" ]; check "fallback ships its own command" $?
|
||||
[ "$fbc" = "claude --model sonnet --effort xhigh --advisor opus" ]; check "fallback ships its own command" $?
|
||||
|
||||
# --- 10. The command carries no 'cd' --------------------------------------
|
||||
# One repo per terminal tab: a startup command prefixed with cd is wrong by
|
||||
|
|
@ -338,7 +340,10 @@ check "no emitted command contains a cd prefix" "$rc"
|
|||
# Model aliases are whatever the INSTALLED claude accepts - never hardcoded
|
||||
# without a gate, because an alias that stops resolving turns every emitted
|
||||
# command into a paste that fails.
|
||||
efforts="$(printf '%s' "$out" | sed -n 's/^command=claude --model [a-z]* --effort //p')"
|
||||
# Capture the effort TOKEN only. Anything may legitimately follow it on the
|
||||
# command line (--advisor does, since section 14), and a match that swallowed
|
||||
# the tail would report a valid effort as invalid.
|
||||
efforts="$(printf '%s' "$out" | sed -n 's/^command=claude --model [a-z]* --effort \([a-z]*\).*/\1/p')"
|
||||
case "|low|medium|high|xhigh|max|" in *"|$efforts|"*) rc=0 ;; *) rc=1 ;; esac
|
||||
check "emitted effort is in the verified effort set" "$rc"
|
||||
|
||||
|
|
@ -409,6 +414,125 @@ check "route.sh no longer claims effort is unobservable from inside" "$rc"
|
|||
grep -q 'CLAUDE_EFFORT' "$R"
|
||||
check "route.sh names the source the caller should measure from" $?
|
||||
|
||||
# --- 14. The advisor: capability gap first, stakes second -----------------
|
||||
# The advisor is a second, stronger model consulted mid-task, so a session
|
||||
# carrying one costs more than a session without. It must therefore fire where
|
||||
# there is a NEED and nowhere else, or it decays into the always-on
|
||||
# advisorModel setting it exists to replace. Two independent needs qualify,
|
||||
# and they turn out to be almost disjoint.
|
||||
#
|
||||
# 1. THE MAIN MODEL IS SONNET (rows 1-2). Here opus is a genuine capability
|
||||
# lift, not a peer: opus judgement at sonnet cost. This is the load-bearing
|
||||
# half, because every fallback-command is one row cheaper and the cheapest
|
||||
# rows are Sonnet - so this is what makes the QUOTA FALLBACK safe to take.
|
||||
# Pinned below by the row-3-falls-back-to-row-2 case.
|
||||
#
|
||||
# 2. A MISTAKE IS EXPENSIVE TO UNDO (reversibility costly|one-way). On an Opus
|
||||
# row the advisor is a peer review rather than a lift, which is worth
|
||||
# paying for when being wrong is not cheap to reverse.
|
||||
#
|
||||
# The two barely overlap: costly forces row 3 and one-way forces row 4, so a
|
||||
# Sonnet row ALWAYS has reversibility=cheap and trigger 2 can never reach it.
|
||||
# Rule 1 covers the Sonnet rows, rule 2 covers the Opus rows.
|
||||
#
|
||||
# What is deliberately NOT a trigger: verification=none on its own. The set it
|
||||
# would add beyond rule 2 is exactly {verification=none AND
|
||||
# reversibility=cheap} - mistakes that are cheap to reverse. It survives on the
|
||||
# Sonnet rows only via rule 1, which is about the model, not the trait.
|
||||
|
||||
adv="$(field known strong cheap local command)"
|
||||
printf '%s' "$adv" | grep -q -- '--advisor opus'
|
||||
check "row 1 (Sonnet/high) always carries an advisor" $?
|
||||
|
||||
adv="$(field known weak cheap local command)"
|
||||
printf '%s' "$adv" | grep -q -- '--advisor opus'
|
||||
check "row 2 (Sonnet/xhigh) always carries an advisor" $?
|
||||
|
||||
adv="$(field known strong costly local command)"
|
||||
printf '%s' "$adv" | grep -q -- '--advisor opus'
|
||||
check "reversibility=costly carries an advisor onto an Opus row" $?
|
||||
|
||||
adv="$(field known strong one-way local command)"
|
||||
printf '%s' "$adv" | grep -q -- '--advisor opus'
|
||||
check "reversibility=one-way carries an advisor onto an Opus row" $?
|
||||
|
||||
# The Opus rows reached WITHOUT expensive stakes are the whole no-advisor set.
|
||||
adv="$(field partial strong cheap local command)"
|
||||
if printf '%s' "$adv" | grep -q -- '--advisor'; then rc=1; else rc=0; fi
|
||||
check "row 3 via path=partial at cheap stakes takes NO advisor" "$rc"
|
||||
|
||||
adv="$(field known strong cheap multi-file command)"
|
||||
if printf '%s' "$adv" | grep -q -- '--advisor'; then rc=1; else rc=0; fi
|
||||
check "row 3 via scope=multi-file at cheap stakes takes NO advisor" "$rc"
|
||||
|
||||
adv="$(field known strong cheap cross-cutting command)"
|
||||
if printf '%s' "$adv" | grep -q -- '--advisor'; then rc=1; else rc=0; fi
|
||||
check "row 4 via scope=cross-cutting at cheap stakes takes NO advisor" "$rc"
|
||||
|
||||
# Fable rejects every advisor (measured against 2.1.220, gated below), so both
|
||||
# triggers must yield to the row. Rows 5 and 6 fire on the failure flag
|
||||
# regardless of reversibility, so this combination is reachable.
|
||||
adv="$("$R" --path known --verification strong --reversibility one-way --scope local \
|
||||
--rationale x --opus-xhigh-failed 2>/dev/null | sed -n 's/^command=//p')"
|
||||
if printf '%s' "$adv" | grep -q -- '--advisor'; then rc=1; else rc=0; fi
|
||||
check "row 5 (Fable) takes no advisor even at one-way stakes" "$rc"
|
||||
|
||||
adv="$("$R" --path undetermined --verification strong --reversibility one-way --scope local \
|
||||
--rationale x --opus-xhigh-failed 2>/dev/null | sed -n 's/^command=//p')"
|
||||
if printf '%s' "$adv" | grep -q -- '--advisor'; then rc=1; else rc=0; fi
|
||||
check "row 6 (Fable) takes no advisor even at one-way stakes" "$rc"
|
||||
|
||||
# The fallback is a real command the operator pastes under quota pressure, so
|
||||
# the advisor decision has to be made per ROW, not once for the winning row.
|
||||
# Row 5 falls back to row 4, which is Opus and CAN advise.
|
||||
adv="$("$R" --path known --verification strong --reversibility one-way --scope local \
|
||||
--rationale x --opus-xhigh-failed 2>/dev/null | sed -n 's/^fallback-command=//p')"
|
||||
printf '%s' "$adv" | grep -q -- '--advisor opus'
|
||||
check "Fable row falling back to an Opus row regains the advisor" $?
|
||||
|
||||
adv="$(field known strong one-way local fallback-command)"
|
||||
printf '%s' "$adv" | grep -q -- '--advisor opus'
|
||||
check "row 4 fallback to row 3 keeps the advisor at one-way stakes" $?
|
||||
|
||||
# THE POINT OF RULE 1, pinned. Row 3 at cheap stakes takes no advisor, but its
|
||||
# quota fallback is row 2 - a Sonnet row - which does. Dropping a row under
|
||||
# quota pressure must not silently drop the safety net with it.
|
||||
adv="$(field partial strong cheap local fallback-command)"
|
||||
printf '%s' "$adv" | grep -q -- '--advisor opus'
|
||||
check "an Opus row with no advisor still falls back to an advised Sonnet row" $?
|
||||
|
||||
adv="$(field known strong cheap cross-cutting fallback-command)"
|
||||
if printf '%s' "$adv" | grep -q -- '--advisor'; then rc=1; else rc=0; fi
|
||||
check "row 4 falling back to row 3 at cheap stakes stays unadvised" "$rc"
|
||||
|
||||
# Closed vocabulary, same reason as the row table: one spelling in circulation.
|
||||
# fable is not merely weaker here, it is REJECTED as an advisor outright.
|
||||
allout="$("$R" --path known --verification strong --reversibility one-way \
|
||||
--scope cross-cutting --rationale x 2>/dev/null)"
|
||||
bad="$(printf '%s' "$allout" | grep -o -- '--advisor [a-z0-9-]*' | grep -v -- '--advisor opus')"
|
||||
[ -z "$bad" ]; check "the only advisor value ever emitted is opus:${bad:- none other}" $?
|
||||
|
||||
if command -v claude >/dev/null 2>&1; then
|
||||
# A real gate that spends NO tokens. Advisor validation runs BEFORE the
|
||||
# empty-prompt check, so `-p ""` reaches the validator and then exits: a
|
||||
# valid advisor fails on the missing prompt, an invalid one fails on itself.
|
||||
# --help cannot gate this - it short-circuits before option validation, so
|
||||
# even an unknown flag exits 0 and the gate would pass without testing.
|
||||
ADVOUT="$(claude --advisor opus -p "" 2>&1 | head -3)"
|
||||
if printf '%s' "$ADVOUT" | grep -q 'unknown option'; then rc=1; else rc=0; fi
|
||||
check "installed claude accepts the --advisor flag at all" "$rc"
|
||||
|
||||
if printf '%s' "$ADVOUT" | grep -q 'cannot be used as an advisor'; then rc=1; else rc=0; fi
|
||||
check "installed claude accepts opus as an advisor" "$rc"
|
||||
|
||||
# The suppression on rows 5 and 6 is load-bearing only while this holds.
|
||||
FABOUT="$(claude --advisor fable -p "" 2>&1 | head -3)"
|
||||
printf '%s' "$FABOUT" | grep -q 'cannot be used as an advisor'
|
||||
check "installed claude still rejects fable as an advisor" $?
|
||||
else
|
||||
echo " skip - claude not on PATH, advisor flag gate not run"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "route-selftest: $PASS passed, $FAIL failed"
|
||||
[ "$FAIL" -eq 0 ] || exit 1
|
||||
|
|
|
|||
|
|
@ -53,6 +53,30 @@
|
|||
# overkill costs quota every session - but a wrong architecture decision in a
|
||||
# published plugin costs more than either.
|
||||
#
|
||||
# THE ADVISOR is emitted into the command as '--advisor opus' - a second,
|
||||
# stronger model consulted at key moments during the session. It is added on a
|
||||
# NEED, never unconditionally: an always-on advisor is the global advisorModel
|
||||
# setting, which burns quota on every session in every repo and is the thing
|
||||
# this rule exists to replace. Two independent needs qualify:
|
||||
#
|
||||
# rows 1-2 ALWAYS. The main model is Sonnet, so opus is a capability LIFT
|
||||
# rather than a peer - opus judgement at sonnet cost. This is what
|
||||
# makes the FALLBACK safe to take: every fallback is one row
|
||||
# cheaper, and the cheapest rows are the Sonnet ones.
|
||||
# rows 3-4 only at reversibility=costly|one-way. The main model is already
|
||||
# Opus, so the advisor buys peer review, worth paying for when a
|
||||
# mistake is not cheap to undo.
|
||||
# rows 5-6 NEVER, and not as a judgement call: the CLI rejects every
|
||||
# advisor for a Fable main model.
|
||||
#
|
||||
# The two triggers barely overlap: costly forces row 3 and one-way forces row
|
||||
# 4, so a Sonnet row always has reversibility=cheap. verification=none is
|
||||
# deliberately not a third trigger - beyond the stakes rule it would only add
|
||||
# mistakes that are cheap to reverse, docs sessions among them.
|
||||
#
|
||||
# Applied per ROW, so 'fallback-command' carries its own correct answer rather
|
||||
# than the winning row's.
|
||||
#
|
||||
# WHERE IT DISAGREES WITH THE RUBRIC'S EXAMPLES. The rows are task-type labels;
|
||||
# the traits are a different classification over the same six outcomes. They
|
||||
# part company by one row on the two cheapest rows - documentation scores
|
||||
|
|
@ -226,7 +250,7 @@ row_name() {
|
|||
5) echo "Fable 5/high" ;; 6) echo "Fable 5/xhigh" ;;
|
||||
esac
|
||||
}
|
||||
row_cmd() {
|
||||
row_base_cmd() {
|
||||
case "$1" in
|
||||
1) echo "claude --model sonnet --effort high" ;;
|
||||
2) echo "claude --model sonnet --effort xhigh" ;;
|
||||
|
|
@ -237,6 +261,39 @@ row_cmd() {
|
|||
esac
|
||||
}
|
||||
|
||||
# THE ADVISOR is a second, stronger model consulted mid-task. It costs real
|
||||
# tokens per session, so it fires on a NEED and nowhere else - an unconditional
|
||||
# advisor is just the global advisorModel setting, which is the thing this
|
||||
# replaces. Two independent needs qualify, and they are almost disjoint:
|
||||
#
|
||||
# rows 1-2 (Sonnet) ALWAYS. opus is a capability LIFT here, not a peer:
|
||||
# opus judgement at sonnet cost. This half is what makes
|
||||
# the fallback-command safe, since every fallback is one
|
||||
# row cheaper and the cheapest rows are Sonnet.
|
||||
# rows 3-4 (Opus) only at costly|one-way stakes, where the advisor is a
|
||||
# peer review and being wrong is not cheap to undo.
|
||||
# rows 5-6 (Fable) never. Not a judgement call: the CLI REJECTS every
|
||||
# advisor for fable ("cannot be used as an advisor"), and
|
||||
# opus is refused as under-capable for a fable main model.
|
||||
# Gated against the installed claude by selftest 14.
|
||||
#
|
||||
# costly forces row 3 and one-way forces row 4, so a Sonnet row always has
|
||||
# reversibility=cheap - the stakes rule can never reach rows 1-2, and the model
|
||||
# rule never reaches rows 3-6. verification=none is deliberately NOT a trigger:
|
||||
# beyond the stakes rule it would only add cheap-to-reverse mistakes, and it
|
||||
# would put an advisor on every docs session (known/none/cheap/local).
|
||||
#
|
||||
# Applied per ROW rather than once, because the fallback is a real command the
|
||||
# operator pastes under quota pressure and must carry its own correct answer.
|
||||
row_advisor() {
|
||||
case "$1" in
|
||||
1|2) echo " --advisor opus" ;;
|
||||
3|4) case "$2" in costly|one-way) echo " --advisor opus" ;; *) echo "" ;; esac ;;
|
||||
*) echo "" ;;
|
||||
esac
|
||||
}
|
||||
row_cmd() { printf '%s%s\n' "$(row_base_cmd "$1")" "$(row_advisor "$1" "$REVERS")"; }
|
||||
|
||||
FB=$((ROW - 1)); [ "$FB" -ge 1 ] || FB=1
|
||||
|
||||
# The rationale is free text from a session and lands inside an HTML comment on
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue