open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 1 Activity

feat(run): a CLI door onto the offline whole-loop run (--scripted-replies)

Browse source 
An adopter without an API budget had two half-doors and no whole one.
`--live-dry-run` takes their own bundle but stops before the first model call
(`run_project` returns a DryRunReport), while `portfolio_optimiser.simulation`
runs the complete loop but only over ITS bundle with ITS scripted answers.
The seam for the missing third case -- the whole loop over your OWN data,
offline -- already existed as `run_project(client_factory=...)` and had zero
CLI exposure. This is the door onto that one seam, not a second implementation
of it (`scripted_factory` is imported lazily; `simulation` imports `run`, so a
module-level import would be circular).

The honesty banner is part of the feature, not decoration (maalbilde §1): a
scripted run that reads like a model run is worse than having no offline mode,
so every scripted invocation prints what is real (context navigation, debate
plumbing, deterministic validator, verdict) and what is not (the answers).

The two offline modes are mutually exclusive rather than one silently winning,
`--report` mode refuses the new flag by allowlist, and a replies file that
cannot serve the run is refused at the door rather than surfacing as a KeyError
mid-run.

Load-bearing MEASURED against the whole suite (645 -> 652), six mutations all
red: detach the wiring · detach the banner · detach the dry-run exclusivity ·
drop the flag from the --report allowlist · make the loader tolerant · control
(print the banner unconditionally).

The --report blade was measured GREEN first: with a non-existent ledger path
the load failure refused before the gate and masked it entirely. Rewritten
against a valid saved ledger, so rc 1 can only come from mode-exclusivity.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GWsexbQjPo9rsV3aUE54ZS
This commit is contained in:
Kjell Tore Guttormsen 2026-08-05 08:55:37 +02:00
commit 3abc61bac3
2 changed files with 247 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

81
src/portfolio_optimiser/run.py
View file

@ -26,8 +26,10 @@ durable learned verdict captured out-of-band in the VerdictStore (D7-portable).
from __future__ import annotations
import asyncio
import json
from collections.abc import Callable, Iterable, Sequence
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Any, Literal, cast
from agent_framework import BaseChatClient, SessionContext
@ -994,6 +996,45 @@ async def run_portfolio(
return base
# The roles ``debate``/``generate`` ask the factory for. Fixed here so a malformed replies file is
# caught at the door instead of mid-run.
_SCRIPTED_ROLES = ("proposer", "checker")
# The honesty banner for the scripted door. It is a REQUIREMENT, not decoration (målbilde §1):
# a scripted run that reads like a model run is worse than having no offline mode at all, so this
# prints on every scripted invocation and mirrors ``simulation.main``'s banner.
_SCRIPTED_BANNER = (
"=" * 78
+ "\nSCRIPTED OFFLINE RUN — every agent reply is read from your --scripted-replies file."
+ "\nNO MODEL WAS CALLED (ingen modellkall gjort). The context navigation, the debate"
+ "\nplumbing, the deterministic validator and the verdict are real; the agents' answers"
+ "\nare yours, not a model's. This proves the loop closes — not that an LLM would say this."
+ "\n"
+ "=" * 78
)
def _load_scripted_replies(path: str) -> dict[str, str]:
"""Load the caller's scripted answers, fail-fast. Every role the debate can ask for must be
present AND a string: a missing role would otherwise surface as a ``KeyError`` deep inside
``scripted_factory``'s lookup, mid-run, long after the run appeared to start cleanly."""
try:
raw = json.loads(Path(path).read_text(encoding="utf-8"))
except FileNotFoundError as exc:
raise ValueError(f"--scripted-replies file not found: {path}") from exc
except json.JSONDecodeError as exc:
raise ValueError(f"--scripted-replies is not valid JSON ({path}): {exc}") from exc
if not isinstance(raw, dict):
raise ValueError(f"--scripted-replies must be a JSON object of role -> reply ({path})")
missing = [r for r in _SCRIPTED_ROLES if not isinstance(raw.get(r), str)]
if missing:
raise ValueError(
f"--scripted-replies needs a string reply for each of {', '.join(_SCRIPTED_ROLES)}; "
f"missing or non-string: {', '.join(missing)} ({path})"
)
return {role: raw[role] for role in _SCRIPTED_ROLES}
def main(argv: list[str] | None = None) -> int:
"""Single-command console entry: run the slice for one project against a docs folder."""
import argparse
@ -1080,6 +1121,16 @@ def main(argv: list[str] | None = None) -> int:
action="store_true",
help="offline drill: build contracts/clients/budget, STOP before the first model call",
)
parser.add_argument(
"--scripted-replies",
default=None,
metavar="FILE",
help="offline WHOLE-LOOP run over your own bundle with ZERO model calls: FILE is JSON "
'{"proposer": "<reply>", "checker": "<reply>"} and those fixed strings stand in for every '
"model answer. Unlike --live-dry-run (which stops before the first call) the complete loop "
"runs — hypothesis, debate, deterministic validator, verdict. The answers are yours, not a "
"model's, and the run says so on every invocation",
)
parser.add_argument(
"--report",
action="store_true",
@ -1123,6 +1174,7 @@ def main(argv: list[str] | None = None) -> int:
"--dimension-config": args.dimension_config is not None,
"--semantic-retrieval": args.semantic_retrieval,
"--embedder-config": args.embedder_config is not None,
"--scripted-replies": args.scripted_replies is not None,
}
if any(report_forbidden.values()):
print(
@ -1254,6 +1306,34 @@ def main(argv: list[str] | None = None) -> int:
)
return 1
# The scripted door (offline WHOLE-loop run over the caller's own bundle). Resolved BEFORE the
# dry-run branch so the two offline modes cannot both be honoured.
scripted_client_factory: Callable[[str], BaseChatClient] | None = None
if args.scripted_replies is not None:
if args.live_dry_run:
# Both are offline, and they contradict: --live-dry-run stops before the first model
# call while --scripted-replies answers every one of them. Refuse rather than let one
# silently win (S5.3's "refused, never ignored" partition).
print(
"run refused: --scripted-replies and --live-dry-run are both offline modes and "
"contradict each other (dry-run stops before the first model call; scripted "
"answers all of them) — pick one",
file=sys.stderr,
)
return 1
try:
replies = _load_scripted_replies(args.scripted_replies)
except (OSError, ValueError) as exc:
print(f"run refused: {exc}", file=sys.stderr)
return 1
# Imported HERE rather than at module scope: ``simulation`` imports ``run``, so a top-level
# import would be circular. The scripted client already exists as MAF-side scaffolding —
# this flag is a DOOR onto that one seam, never a second implementation of it.
from portfolio_optimiser.simulation import scripted_factory
scripted_client_factory = scripted_factory(replies, [])
print(_SCRIPTED_BANNER)
if args.live_dry_run:
# S4.2 drill (comparison protocol §4 pkt 2/3): walk the offline path, STOP before the first
# model call, print the run-config. A misconfigured profile (e.g. AZURE with a
@ -1325,6 +1405,7 @@ def main(argv: list[str] | None = None) -> int:
run_id=args.run_id,
verdict_input={"decision": args.decision, "rationale": args.rationale},
semantic_retrieval=args.semantic_retrieval,
client_factory=scripted_client_factory,
)
),
)