"""The portfolio CLI's two silent surfaces, measured from a fresh clone (adoption walk 2/2). Walking ``--portfolio`` end-to-end as a downloader would — which no session had done; the 2026-08-03 measurement found ZERO reachings of ``run_portfolio`` in the simulation — turned up two defects of the SAME class the repo already legislates against elsewhere, and neither was reachable by a test that never invoked the CLI's portfolio branch: **(A) ``--scripted-replies`` is silently DROPPED in portfolio mode.** ``main()``'s portfolio dispatch returns (``run.py``, the ``if args.portfolio:`` block) BEFORE the scripted-replies block that builds the client factory, and the flag is absent from the ``single_only`` refusal set — so the flag neither takes effect nor is refused, the honesty banner never prints, and the pass attempts REAL model calls. MEASURED against the shipped reference portfolio: four projects, four ``APIConnectionError`` failures. This is precisely the "refused, never ignored" partition (S5.3) the ``--report`` allowlist enforces for this very flag — the previous session joined the flag to one partition and missed the other. The fix WIRES it rather than refusing it: ``run_portfolio`` already exposes ``client_factory``, the same seam ``--scripted-replies`` was built to expose on the single-project path, and refusing would leave portfolio mode with no offline door at all for an adopter without a model budget. **(B) A portfolio pass reports only ONE of its four outcome channels.** ``main()`` printed ``runs`` and ``stop_reason`` and nothing else: ``failures`` (S3.3 collect-and-continue) and ``budget_stop`` (S3.4 global cap) never reached the operator, and rc was unconditionally 0. MEASURED: the four-failure pass above printed NOTHING AT ALL and exited 0 — silence read as success. ``BudgetStop``'s own docstring argues that folding exhaustion into ``stop_reason`` "would let a caller read 'we stopped' without being able to tell which happened"; the CLI showed neither. The rc rule is ``failures`` non-empty -> 1: collect-and-continue exists so a partial pass does not LOSE the completed work (which still prints on stdout), not so a pass with dead projects can call itself a success. Load-bearing (each blade detaches exactly one thing): 1. the scripted factory reaches ``run_portfolio`` -> a REAL offline portfolio pass (RED when the ``client_factory=`` wiring is dropped: the pass falls back to the production factory); 2. that pass is genuinely model-free (RED if a real factory is built); 3. the banner prints in portfolio mode too (RED when the banner is moved back below the dispatch); 4. control: no banner without the flag, so blade 3 cannot pass on a constant; 5. failures are visible AND carry their own values (RED when the failure print is detached); 6. rc reflects them (RED when rc stays 0); 7. control: a clean pass prints no failure line and exits 0 — so 5/6 cannot pass on a constant; 8. a completed run still prints alongside a failure (collect-and-continue is not undone by 6); 9. the budget stop is visible with its own numbers (RED when that print is detached); 10. control: no budget-stop line when the pass was not budget-stopped. """ from __future__ import annotations import json from pathlib import Path import pytest from portfolio_optimiser import run from portfolio_optimiser.ledger import LedgerEntry, SavingsLedger from portfolio_optimiser.run import BudgetStop, PortfolioResult, RunFailure from portfolio_optimiser.verdicts import VerdictStore # The same caller-supplied answers the single-project door documents. On the shipped ROAD # portfolio these are rejected by S4.0's baseline anchoring (the cost code belongs to the bygg # bundle, not to a road project's estimate) — which is the correct outcome and is beside the # point here: the blades below assert that the pass RUNS offline, not that it validates. _VALID_PROPOSAL = ( '{"measure":"LED-retrofit av kontorbelysning","affected_items":' '[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}' ) _CHECKER_APPROVE = "Tallene er innenfor feasibelt område og resonnementet holder. VERDICT: APPROVE" @pytest.fixture(autouse=True) def _isolate_model_env(monkeypatch: pytest.MonkeyPatch) -> None: """Hermetic env (mirrors ``test_scripted_cli_door_loadbearing.py``).""" monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False) monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False) @pytest.fixture() def replies_file(tmp_path: Path) -> Path: path = tmp_path / "replies.json" path.write_text( json.dumps({"proposer": _VALID_PROPOSAL, "checker": _CHECKER_APPROVE}), encoding="utf-8", ) return path # -------------------------------------------------------------------------------------------- # (A) the offline door reaches portfolio mode # -------------------------------------------------------------------------------------------- def test_portfolio_runs_offline_with_scripted_replies(replies_file, capsys) -> None: """Blade 1 — ``--portfolio --scripted-replies`` completes a real pass over the shipped reference portfolio with zero model calls. RED before the wiring: every project raises ``APIConnectionError`` (measured: 4 runs, 4 failures) so no outcome line is printed.""" rc = run.main(["--portfolio", "--scripted-replies", str(replies_file)]) out = capsys.readouterr().out assert rc == 0, out # One outcome line per project in the shipped portfolio (4), each carrying a minted verdict id. assert out.count("verdict id=") == 4, out def test_portfolio_scripted_pass_makes_no_real_client(replies_file, monkeypatch, capsys) -> None: """Blade 2 — genuinely model-free, and the ONE blade here that proves defect A on its own. It asserts on a CALL SENTINEL, not on an exception. The first version of this blade raised from the patched factory, and that version was measured GREEN with both the wiring and the rc rule detached: ``_default_factory`` is called inside ``run_project``'s coroutine (``run.py``), and ``run_portfolio`` gathers with ``return_exceptions=True``, so the ``AssertionError`` was collected into a ``RunFailure`` and never escaped — the blade only went red because rc had become 1, i.e. it was testing defect B while claiming to test defect A. A list appended in the factory and asserted in the test body cannot be swallowed by the wave handler. This matters because blades 1 and 8 are ENVIRONMENT-CONDITIONAL: the local profile points at loopback, so on a machine actually running a local model server the A-detach would make real calls and could complete the pass. Their red is real here and was measured here, but it is not portable. This blade's red is.""" built: list[str] = [] def _record(profile: str): built.append(profile) raise AssertionError("a real client factory was built on the scripted portfolio path") monkeypatch.setattr(run, "_default_factory", _record) rc = run.main(["--portfolio", "--scripted-replies", str(replies_file)]) out = capsys.readouterr().out assert built == [], f"the production factory was built {len(built)} time(s): {built}" assert rc == 0, out def test_portfolio_scripted_pass_says_so_unmistakably(replies_file, capsys) -> None: """Blade 3 — the honesty banner is not a single-project-mode courtesy. A scripted portfolio pass that reads like a model pass is the same failure mode (målbilde §1).""" run.main(["--portfolio", "--scripted-replies", str(replies_file)]) out = capsys.readouterr().out.upper() assert "SCRIPTED" in out assert "NO MODEL" in out or "INGEN MODELLKALL" in out def test_no_banner_in_portfolio_mode_without_the_flag(tmp_path, capsys) -> None: """Blade 4 (control) — the banner must be CAUSED by the flag. Uses a met portfolio goal so the control stops offline at the goal check, before any client is built.""" goals = tmp_path / "goals.json" goals.write_text('{"portfolio": {"absolute_ore": 1, "mode": "hard"}}', encoding="utf-8") led = SavingsLedger() led.add_realized( LedgerEntry( project_id="FV42-GSV-E1", dimension="energi", candidate_identity="c1", amount_ore=1, verdict_id="v1", provenance="x", ) ) ledger = tmp_path / "ledger.json" led.save(str(ledger)) rc = run.main(["--portfolio", "--goals", str(goals), "--ledger", str(ledger)]) assert rc == 0 assert "SCRIPTED" not in capsys.readouterr().out.upper() # -------------------------------------------------------------------------------------------- # (B) the silent outcome channels # -------------------------------------------------------------------------------------------- def _result( *, runs: tuple = (), failures: tuple[RunFailure, ...] = (), budget_stop: BudgetStop | None = None, ) -> PortfolioResult: """A crafted ``PortfolioResult`` — the unit under test is ``main()``'s reporting block, and driving a genuine mid-pass failure through the CLI is impossible offline (the CLI builds its own client factory from the replies file, so no failing client can be injected).""" return PortfolioResult( runs=runs, store=VerdictStore([]), validated_count=0, rejected_count=0, sum_claimed_saving_nok=0.0, sum_token_usage=0, stopped_early=budget_stop is not None, failures=failures, budget_stop=budget_stop, ) @pytest.fixture() def stub_portfolio(monkeypatch): """Install a ``run_portfolio`` stub returning a chosen ``PortfolioResult``.""" def _install(result: PortfolioResult) -> None: async def _fake(*_args, **_kwargs) -> PortfolioResult: return result monkeypatch.setattr(run, "run_portfolio", _fake) return _install def test_failures_are_visible_with_their_own_values(stub_portfolio, capsys) -> None: """Blade 5 — a project that RAISED must reach the operator. TWO failures with distinct ids and distinct messages: a canned string cannot produce both, so this cannot pass on a constant. RED before the fix: the pass printed nothing at all.""" stub_portfolio( _result( failures=( RunFailure("FV42-GSV-E1", "Connection error.", "APIConnectionError"), RunFailure("RV13-RAS-TP", "budget blew up", "BudgetExceeded"), ) ) ) rc = run.main(["--portfolio"]) err = capsys.readouterr().err assert rc == 1 assert "FV42-GSV-E1" in err and "Connection error." in err assert "RV13-RAS-TP" in err and "budget blew up" in err assert "APIConnectionError" in err and "BudgetExceeded" in err def test_a_pass_with_failures_does_not_exit_zero(stub_portfolio, capsys) -> None: """Blade 6 — silence-as-success was the worse half of the defect: a scripted caller checking only rc learned nothing. MEASURED before the fix: four dead projects, rc 0.""" stub_portfolio(_result(failures=(RunFailure("FV42-GSV-E1", "boom", "RuntimeError"),))) assert run.main(["--portfolio"]) == 1 capsys.readouterr() def test_a_clean_pass_stays_silent_and_exits_zero(stub_portfolio, capsys) -> None: """Blade 7 (control) — no failures means no failure line and rc 0, so blades 5/6 cannot pass on an unconditional print or an unconditional rc.""" stub_portfolio(_result()) rc = run.main(["--portfolio"]) captured = capsys.readouterr() assert rc == 0 assert "failed" not in captured.err.lower() assert captured.err.strip() == "" def test_completed_runs_still_print_alongside_a_failure(replies_file, monkeypatch, capsys) -> None: """Blade 8 — the non-zero rc must not undo collect-and-continue: the projects that COMPLETED still report their outcome on stdout. Driven through the real scripted pass, then one crafted failure appended, so the ``runs`` side is genuine and not a stub artefact.""" real_run_portfolio = run.run_portfolio async def _fake(*args, **kwargs) -> PortfolioResult: genuine = await real_run_portfolio(*args, **kwargs) return _result( runs=genuine.runs, failures=(RunFailure("RV13-RAS-TP", "boom", "RuntimeError"),), ) monkeypatch.setattr(run, "run_portfolio", _fake) rc = run.main(["--portfolio", "--scripted-replies", str(replies_file)]) captured = capsys.readouterr() assert rc == 1 assert captured.out.count("verdict id=") == 4, captured.out assert "RV13-RAS-TP" in captured.err def test_budget_stop_is_visible_with_its_own_numbers(stub_portfolio, capsys) -> None: """Blade 9 — the S3.4 global-cap stop is a distinct outcome from a goal stop (``BudgetStop`` is a separate field for exactly that reason) and was equally invisible. The four numbers are asserted individually: their DIFFERENCE is the operator's next decision.""" stub_portfolio( _result( budget_stop=BudgetStop( limit_tokens=500, spent_tokens=470, remaining_tokens=30, required_tokens=120 ) ) ) rc = run.main(["--portfolio"]) out = capsys.readouterr().out assert rc == 0 # exhaustion is a structured stop, not a failure — it does not fail the pass assert "budget" in out.lower() for number in ("500", "470", "30", "120"): assert number in out, out def test_no_budget_stop_line_when_the_pass_was_not_stopped(stub_portfolio, capsys) -> None: """Blade 10 (control) — blade 9 cannot pass on an unconditional line.""" stub_portfolio(_result()) run.main(["--portfolio"]) assert "budget" not in capsys.readouterr().out.lower()