fix(run): the portfolio CLI stops swallowing its offline door and its failures
Walking --portfolio end-to-end as a downloader would -- which no session had done -- surfaced two defects of a class this repo already legislates against. (A) --scripted-replies was silently DROPPED in portfolio mode. main()'s portfolio dispatch returned before the block that builds the scripted client factory, and the flag was absent from the single_only refusal set: neither honoured nor refused. Measured against the shipped reference portfolio: no banner, four real model calls attempted, four APIConnectionError. The previous session joined this flag to the --report allowlist and missed the portfolio partition. Resolved by WIRING rather than refusing -- run_portfolio already exposes the same client_factory seam, and refusing would have left portfolio mode with no offline door at all for an adopter without a model budget. The scripted block is hoisted above the dispatch; the single-project required-arg and semantic-retrieval refusals are hoisted with it so an incomplete argv is still refused BEFORE the honesty banner could claim a scripted run happened, and the refusal order within single-project mode is unchanged. (B) A portfolio pass reported one of its four outcome channels. failures (S3.3 collect-and-continue) and budget_stop (S3.4 global cap) never reached the operator and rc was unconditionally 0, so the four-failure pass above printed NOTHING and exited 0 -- silence read as success. BudgetStop is a separate field precisely so exhaustion can be told from success; the CLI showed neither. Failures now print to stderr with project id, error type and message; the budget stop prints its four numbers; rc is 1 iff something raised. A budget stop alone stays rc 0: exhaustion is a structured stop the operator asked for by setting a cap, not a crash. Completed runs still print, so the non-zero rc does not undo collect-and-continue. Load-bearing MEASURED against the whole 662-test suite, seven mutations all red, including both controls: detach the client_factory wiring - make the banner a single-project courtesy again - detach the failure print - revert rc to 0 - detach the budget-stop print - print the failure line unconditionally (control) - print the budget-stop line unconditionally (control). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0118noV9rCfrdREH26XqZB5z
This commit is contained in:
parent
392f8493da
commit
415ebbb7f2
2 changed files with 344 additions and 44 deletions
|
|
@ -1230,6 +1230,81 @@ def main(argv: list[str] | None = None) -> int:
|
|||
)
|
||||
return 1
|
||||
|
||||
# Single-project mode requires PROJECT_ID + --docs-dir (compensating for the relaxed argparse
|
||||
# required/positional so the legacy contract keeps failing loudly via the refusal surface).
|
||||
# HOISTED above the scripted door (below) so an incomplete argv is refused BEFORE the honesty
|
||||
# banner could claim a scripted run happened; the refusal ORDER within single-project mode
|
||||
# (required args -> semantic-retrieval -> scripted) is unchanged.
|
||||
if not args.portfolio and (args.project_id is None or args.docs_dir is None):
|
||||
print(
|
||||
"run refused: single-project mode requires PROJECT_ID and --docs-dir "
|
||||
"(use --portfolio for portfolio mode)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
# --semantic-retrieval is refused, never silently ignored (the repo's flag contract). In
|
||||
# single-project mode it can only do observable work with BOTH of these: the Step-1 fold is
|
||||
# gated on ``bundle_dir``, and ``--verdict-dir`` is the only route by which ``main()`` can hand
|
||||
# ``run_project`` a non-empty store (``main()`` never passes ``store=``, and ``run_project``
|
||||
# never seeds one). Without them the flag would rank nothing that reaches a prompt, and
|
||||
# ``RunResult.retrieved`` never leaves the process — ``main()`` prints one outcome line only.
|
||||
#
|
||||
# DELIBERATELY STATIC. There is no runtime "refuse if the store ends up empty" check: a
|
||||
# missing, empty or partially-skipped inbox is the Steg-7 tolerant-load contract, so refusing
|
||||
# there would fire on a legitimate first run. The refusal is therefore necessary, not
|
||||
# sufficient — it catches the configuration that CANNOT work, not every run that finds nothing.
|
||||
#
|
||||
# main() only. As a library API, ``run_project(semantic_retrieval=True, store=…)`` with a
|
||||
# caller-supplied store stays legitimate — that is the path the tests drive. Portfolio mode is
|
||||
# unaffected: ``run_portfolio`` always resolves a store and populates it by cross-project capture.
|
||||
if not args.portfolio and args.semantic_retrieval:
|
||||
required = {"--bundle-dir": args.bundle_dir, "--verdict-dir": args.verdict_dir}
|
||||
missing = [name for name, value in required.items() if not value]
|
||||
if missing:
|
||||
print(
|
||||
f"run refused: --semantic-retrieval requires {' and '.join(missing)} in "
|
||||
"single-project mode (the Step-1 fold is bundle-path-only, and --verdict-dir is "
|
||||
"the only route to a non-empty store)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
# The scripted door (offline WHOLE-loop run over the caller's own data). Resolved BEFORE the
|
||||
# dry-run branch so the two offline modes cannot both be honoured — and BEFORE the portfolio
|
||||
# dispatch, because the door serves BOTH modes. It originally sat below that dispatch, which
|
||||
# made ``--portfolio --scripted-replies`` silently drop the flag: no banner, and four real
|
||||
# model calls attempted (measured). That is the failure mode the "refused, never ignored"
|
||||
# partition exists to prevent, and here the honest resolution is to WIRE it — ``run_portfolio``
|
||||
# already exposes the same ``client_factory`` seam ``run_project`` does, so refusing would have
|
||||
# left portfolio mode with no offline door at all for an adopter without a model budget.
|
||||
scripted_client_factory: Callable[[str], BaseChatClient] | None = None
|
||||
if args.scripted_replies is not None:
|
||||
if args.live_dry_run:
|
||||
# Both are offline, and they contradict: --live-dry-run stops before the first model
|
||||
# call while --scripted-replies answers every one of them. Refuse rather than let one
|
||||
# silently win (S5.3's "refused, never ignored" partition). Unreachable in portfolio
|
||||
# mode, where --live-dry-run is already refused by the single_only partition above.
|
||||
print(
|
||||
"run refused: --scripted-replies and --live-dry-run are both offline modes and "
|
||||
"contradict each other (dry-run stops before the first model call; scripted "
|
||||
"answers all of them) — pick one",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
try:
|
||||
replies = _load_scripted_replies(args.scripted_replies)
|
||||
except (OSError, ValueError) as exc:
|
||||
print(f"run refused: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
# Imported HERE rather than at module scope: ``simulation`` imports ``run``, so a top-level
|
||||
# import would be circular. The scripted client already exists as MAF-side scaffolding —
|
||||
# this flag is a DOOR onto that one seam, never a second implementation of it.
|
||||
from portfolio_optimiser.simulation import scripted_factory
|
||||
|
||||
scripted_client_factory = scripted_factory(replies, [])
|
||||
print(_SCRIPTED_BANNER)
|
||||
|
||||
if args.portfolio:
|
||||
# Portfolio mode (Step 3): dispatch to the EXISTING run_portfolio via the fail-fast loaders
|
||||
# (run_portfolio itself is unchanged). Loader/ValueError failures surface through the same
|
||||
|
|
@ -1253,6 +1328,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
ledger=ledger,
|
||||
goals=goals,
|
||||
semantic_retrieval=args.semantic_retrieval,
|
||||
client_factory=scripted_client_factory,
|
||||
)
|
||||
)
|
||||
except (ValueError, FileNotFoundError, ValidationError) as exc:
|
||||
|
|
@ -1267,72 +1343,29 @@ def main(argv: list[str] | None = None) -> int:
|
|||
f"observed_ore={sr.observed_ore} limit_ore={sr.limit_ore} "
|
||||
f"stopped_early={portfolio_result.stopped_early}"
|
||||
)
|
||||
return 0
|
||||
|
||||
# Single-project mode requires PROJECT_ID + --docs-dir (compensating for the relaxed argparse
|
||||
# required/positional so the legacy contract keeps failing loudly via the refusal surface).
|
||||
if args.project_id is None or args.docs_dir is None:
|
||||
print(
|
||||
"run refused: single-project mode requires PROJECT_ID and --docs-dir "
|
||||
"(use --portfolio for portfolio mode)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
# --semantic-retrieval is refused, never silently ignored (the repo's flag contract). In
|
||||
# single-project mode it can only do observable work with BOTH of these: the Step-1 fold is
|
||||
# gated on ``bundle_dir``, and ``--verdict-dir`` is the only route by which ``main()`` can hand
|
||||
# ``run_project`` a non-empty store (``main()`` never passes ``store=``, and ``run_project``
|
||||
# never seeds one). Without them the flag would rank nothing that reaches a prompt, and
|
||||
# ``RunResult.retrieved`` never leaves the process — ``main()`` prints one outcome line only.
|
||||
#
|
||||
# DELIBERATELY STATIC. There is no runtime "refuse if the store ends up empty" check: a
|
||||
# missing, empty or partially-skipped inbox is the Steg-7 tolerant-load contract, so refusing
|
||||
# there would fire on a legitimate first run. The refusal is therefore necessary, not
|
||||
# sufficient — it catches the configuration that CANNOT work, not every run that finds nothing.
|
||||
#
|
||||
# main() only. As a library API, ``run_project(semantic_retrieval=True, store=…)`` with a
|
||||
# caller-supplied store stays legitimate — that is the path the tests drive. Portfolio mode is
|
||||
# unaffected: ``run_portfolio`` always resolves a store and populates it by cross-project capture.
|
||||
if args.semantic_retrieval:
|
||||
required = {"--bundle-dir": args.bundle_dir, "--verdict-dir": args.verdict_dir}
|
||||
missing = [name for name, value in required.items() if not value]
|
||||
if missing:
|
||||
# A ``PortfolioResult`` has FOUR outcome channels and this branch reported one of them:
|
||||
# ``failures`` and ``budget_stop`` never reached the operator, and rc was unconditionally 0
|
||||
# — so a pass in which every project died printed nothing and exited 0 (measured: four
|
||||
# projects, four APIConnectionError, silent success). ``BudgetStop`` is kept apart from
|
||||
# ``stop_reason`` precisely so a caller can tell exhaustion from success; showing neither
|
||||
# collapsed the distinction the dataclass was split to preserve.
|
||||
if portfolio_result.budget_stop is not None:
|
||||
bs = portfolio_result.budget_stop
|
||||
print(
|
||||
f"run refused: --semantic-retrieval requires {' and '.join(missing)} in "
|
||||
"single-project mode (the Step-1 fold is bundle-path-only, and --verdict-dir is "
|
||||
"the only route to a non-empty store)",
|
||||
f"budget stop: limit_tokens={bs.limit_tokens} spent_tokens={bs.spent_tokens} "
|
||||
f"remaining_tokens={bs.remaining_tokens} required_tokens={bs.required_tokens} "
|
||||
f"stopped_early={portfolio_result.stopped_early}"
|
||||
)
|
||||
for failure in portfolio_result.failures:
|
||||
print(
|
||||
f"project failed: {failure.project_id} [{failure.error_type}] {failure.error}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
# The scripted door (offline WHOLE-loop run over the caller's own bundle). Resolved BEFORE the
|
||||
# dry-run branch so the two offline modes cannot both be honoured.
|
||||
scripted_client_factory: Callable[[str], BaseChatClient] | None = None
|
||||
if args.scripted_replies is not None:
|
||||
if args.live_dry_run:
|
||||
# Both are offline, and they contradict: --live-dry-run stops before the first model
|
||||
# call while --scripted-replies answers every one of them. Refuse rather than let one
|
||||
# silently win (S5.3's "refused, never ignored" partition).
|
||||
print(
|
||||
"run refused: --scripted-replies and --live-dry-run are both offline modes and "
|
||||
"contradict each other (dry-run stops before the first model call; scripted "
|
||||
"answers all of them) — pick one",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
try:
|
||||
replies = _load_scripted_replies(args.scripted_replies)
|
||||
except (OSError, ValueError) as exc:
|
||||
print(f"run refused: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
# Imported HERE rather than at module scope: ``simulation`` imports ``run``, so a top-level
|
||||
# import would be circular. The scripted client already exists as MAF-side scaffolding —
|
||||
# this flag is a DOOR onto that one seam, never a second implementation of it.
|
||||
from portfolio_optimiser.simulation import scripted_factory
|
||||
|
||||
scripted_client_factory = scripted_factory(replies, [])
|
||||
print(_SCRIPTED_BANNER)
|
||||
# rc 1 iff something RAISED. Collect-and-continue (S3.3) exists so a partial pass does not
|
||||
# LOSE the work that completed — every finished run still printed above — not so a pass with
|
||||
# dead projects can report success to a scripted caller. A ``budget_stop`` alone stays rc 0:
|
||||
# exhaustion is a structured stop the operator asked for by setting a cap, not a crash.
|
||||
return 1 if portfolio_result.failures else 0
|
||||
|
||||
if args.live_dry_run:
|
||||
# S4.2 drill (comparison protocol §4 pkt 2/3): walk the offline path, STOP before the first
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue