"""The portfolio CLI's two silent surfaces, measured from a fresh clone (adoption walk 2/2). Walking ``--portfolio`` end-to-end as a downloader would — which no session had done; the 2026-08-03 measurement found ZERO reachings of ``run_portfolio`` in the simulation — turned up two defects of the SAME class the repo already legislates against elsewhere, and neither was reachable by a test that never invoked the CLI's portfolio branch: **(A) ``--scripted-replies`` is silently DROPPED in portfolio mode.** ``main()``'s portfolio dispatch returns (``run.py``, the ``if args.portfolio:`` block) BEFORE the scripted-replies block that builds the client factory, and the flag is absent from the ``single_only`` refusal set — so the flag neither takes effect nor is refused, the honesty banner never prints, and the pass attempts REAL model calls. MEASURED against the shipped reference portfolio: four projects, four ``APIConnectionError`` failures. This is precisely the "refused, never ignored" partition (S5.3) the ``--report`` allowlist enforces for this very flag — the previous session joined the flag to one partition and missed the other. The fix WIRES it rather than refusing it: ``run_portfolio`` already exposes ``client_factory``, the same seam ``--scripted-replies`` was built to expose on the single-project path, and refusing would leave portfolio mode with no offline door at all for an adopter without a model budget. **(B) A portfolio pass reports only ONE of its four outcome channels.** ``main()`` printed ``runs`` and ``stop_reason`` and nothing else: ``failures`` (S3.3 collect-and-continue) and ``budget_stop`` (S3.4 global cap) never reached the operator, and rc was unconditionally 0. MEASURED: the four-failure pass above printed NOTHING AT ALL and exited 0 — silence read as success. ``BudgetStop``'s own docstring argues that folding exhaustion into ``stop_reason`` "would let a caller read 'we stopped' without being able to tell which happened"; the CLI showed neither. The rc rule is ``failures`` non-empty -> 1: collect-and-continue exists so a partial pass does not LOSE the completed work (which still prints on stdout), not so a pass with dead projects can call itself a success. Load-bearing (each blade detaches exactly one thing): 1. the scripted factory reaches ``run_portfolio`` -> a REAL offline portfolio pass (RED when the ``client_factory=`` wiring is dropped: the pass falls back to the production factory); 2. that pass is genuinely model-free (RED if a real factory is built); 3. the banner prints in portfolio mode too (RED when the banner is moved back below the dispatch); 4. control: no banner without the flag, so blade 3 cannot pass on a constant; 5. failures are visible AND carry their own values (RED when the failure print is detached); 6. rc reflects them (RED when rc stays 0); 7. control: a clean pass prints no failure line and exits 0 — so 5/6 cannot pass on a constant; 8. a completed run still prints alongside a failure (collect-and-continue is not undone by 6); 9. the budget stop is visible with its own numbers (RED when that print is detached); 10. control: no budget-stop line when the pass was not budget-stopped. """ from __future__ import annotations import json from pathlib import Path import pytest from portfolio_optimiser import run from portfolio_optimiser.ledger import LedgerEntry, SavingsLedger from portfolio_optimiser.run import BudgetStop, PortfolioResult, RunFailure from portfolio_optimiser.verdicts import VerdictStore # The same caller-supplied answers the single-project door documents. On the shipped ROAD # portfolio these are rejected by S4.0's baseline anchoring (the cost code belongs to the bygg # bundle, not to a road project's estimate) — which is the correct outcome and is beside the # point here: the blades below assert that the pass RUNS offline, not that it validates. _VALID_PROPOSAL = ( '{"measure":"LED-retrofit av kontorbelysning","affected_items":' '[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}' ) _CHECKER_APPROVE = "Tallene er innenfor feasibelt område og resonnementet holder. VERDICT: APPROVE" @pytest.fixture(autouse=True) def _isolate_model_env(monkeypatch: pytest.MonkeyPatch) -> None: """Hermetic env (mirrors ``test_scripted_cli_door_loadbearing.py``).""" monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False) monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False) @pytest.fixture() def replies_file(tmp_path: Path) -> Path: path = tmp_path / "replies.json" path.write_text( json.dumps({"proposer": _VALID_PROPOSAL, "checker": _CHECKER_APPROVE}), encoding="utf-8", ) return path # -------------------------------------------------------------------------------------------- # (A) the offline door reaches portfolio mode # -------------------------------------------------------------------------------------------- def test_portfolio_runs_offline_with_scripted_replies(replies_file, capsys) -> None: """Blade 1 — ``--portfolio --scripted-replies`` completes a real pass over the shipped reference portfolio with zero model calls. RED before the wiring: every project raises ``APIConnectionError`` (measured: 4 runs, 4 failures) so no outcome line is printed.""" rc = run.main(["--portfolio", "--scripted-replies", str(replies_file)]) out = capsys.readouterr().out assert rc == 0, out # One outcome line per project in the shipped portfolio (4), each carrying a minted verdict id. assert out.count("verdict id=") == 4, out def test_portfolio_scripted_pass_makes_no_real_client(replies_file, monkeypatch, capsys) -> None: """Blade 2 — genuinely model-free. Detonates if the portfolio dispatch falls back to the production factory (which is exactly what the un-wired flag did).""" def _boom(_profile: str): # pragma: no cover - the point is that it never runs raise AssertionError("a real client factory was built on the scripted portfolio path") monkeypatch.setattr(run, "_default_factory", _boom) rc = run.main(["--portfolio", "--scripted-replies", str(replies_file)]) assert rc == 0, capsys.readouterr().out def test_portfolio_scripted_pass_says_so_unmistakably(replies_file, capsys) -> None: """Blade 3 — the honesty banner is not a single-project-mode courtesy. A scripted portfolio pass that reads like a model pass is the same failure mode (målbilde §1).""" run.main(["--portfolio", "--scripted-replies", str(replies_file)]) out = capsys.readouterr().out.upper() assert "SCRIPTED" in out assert "NO MODEL" in out or "INGEN MODELLKALL" in out def test_no_banner_in_portfolio_mode_without_the_flag(tmp_path, capsys) -> None: """Blade 4 (control) — the banner must be CAUSED by the flag. Uses a met portfolio goal so the control stops offline at the goal check, before any client is built.""" goals = tmp_path / "goals.json" goals.write_text('{"portfolio": {"absolute_ore": 1, "mode": "hard"}}', encoding="utf-8") led = SavingsLedger() led.add_realized( LedgerEntry( project_id="FV42-GSV-E1", dimension="energi", candidate_identity="c1", amount_ore=1, verdict_id="v1", provenance="x", ) ) ledger = tmp_path / "ledger.json" led.save(str(ledger)) rc = run.main(["--portfolio", "--goals", str(goals), "--ledger", str(ledger)]) assert rc == 0 assert "SCRIPTED" not in capsys.readouterr().out.upper() # -------------------------------------------------------------------------------------------- # (B) the silent outcome channels # -------------------------------------------------------------------------------------------- def _result( *, runs: tuple = (), failures: tuple[RunFailure, ...] = (), budget_stop: BudgetStop | None = None, ) -> PortfolioResult: """A crafted ``PortfolioResult`` — the unit under test is ``main()``'s reporting block, and driving a genuine mid-pass failure through the CLI is impossible offline (the CLI builds its own client factory from the replies file, so no failing client can be injected).""" return PortfolioResult( runs=runs, store=VerdictStore([]), validated_count=0, rejected_count=0, sum_claimed_saving_nok=0.0, sum_token_usage=0, stopped_early=budget_stop is not None, failures=failures, budget_stop=budget_stop, ) @pytest.fixture() def stub_portfolio(monkeypatch): """Install a ``run_portfolio`` stub returning a chosen ``PortfolioResult``.""" def _install(result: PortfolioResult) -> None: async def _fake(*_args, **_kwargs) -> PortfolioResult: return result monkeypatch.setattr(run, "run_portfolio", _fake) return _install def test_failures_are_visible_with_their_own_values(stub_portfolio, capsys) -> None: """Blade 5 — a project that RAISED must reach the operator. TWO failures with distinct ids and distinct messages: a canned string cannot produce both, so this cannot pass on a constant. RED before the fix: the pass printed nothing at all.""" stub_portfolio( _result( failures=( RunFailure("FV42-GSV-E1", "Connection error.", "APIConnectionError"), RunFailure("RV13-RAS-TP", "budget blew up", "BudgetExceeded"), ) ) ) rc = run.main(["--portfolio"]) err = capsys.readouterr().err assert rc == 1 assert "FV42-GSV-E1" in err and "Connection error." in err assert "RV13-RAS-TP" in err and "budget blew up" in err assert "APIConnectionError" in err and "BudgetExceeded" in err def test_a_pass_with_failures_does_not_exit_zero(stub_portfolio, capsys) -> None: """Blade 6 — silence-as-success was the worse half of the defect: a scripted caller checking only rc learned nothing. MEASURED before the fix: four dead projects, rc 0.""" stub_portfolio(_result(failures=(RunFailure("FV42-GSV-E1", "boom", "RuntimeError"),))) assert run.main(["--portfolio"]) == 1 capsys.readouterr() def test_a_clean_pass_stays_silent_and_exits_zero(stub_portfolio, capsys) -> None: """Blade 7 (control) — no failures means no failure line and rc 0, so blades 5/6 cannot pass on an unconditional print or an unconditional rc.""" stub_portfolio(_result()) rc = run.main(["--portfolio"]) captured = capsys.readouterr() assert rc == 0 assert "failed" not in captured.err.lower() assert captured.err.strip() == "" def test_completed_runs_still_print_alongside_a_failure(replies_file, monkeypatch, capsys) -> None: """Blade 8 — the non-zero rc must not undo collect-and-continue: the projects that COMPLETED still report their outcome on stdout. Driven through the real scripted pass, then one crafted failure appended, so the ``runs`` side is genuine and not a stub artefact.""" real_run_portfolio = run.run_portfolio async def _fake(*args, **kwargs) -> PortfolioResult: genuine = await real_run_portfolio(*args, **kwargs) return _result( runs=genuine.runs, failures=(RunFailure("RV13-RAS-TP", "boom", "RuntimeError"),), ) monkeypatch.setattr(run, "run_portfolio", _fake) rc = run.main(["--portfolio", "--scripted-replies", str(replies_file)]) captured = capsys.readouterr() assert rc == 1 assert captured.out.count("verdict id=") == 4, captured.out assert "RV13-RAS-TP" in captured.err def test_budget_stop_is_visible_with_its_own_numbers(stub_portfolio, capsys) -> None: """Blade 9 — the S3.4 global-cap stop is a distinct outcome from a goal stop (``BudgetStop`` is a separate field for exactly that reason) and was equally invisible. The four numbers are asserted individually: their DIFFERENCE is the operator's next decision.""" stub_portfolio( _result( budget_stop=BudgetStop( limit_tokens=500, spent_tokens=470, remaining_tokens=30, required_tokens=120 ) ) ) rc = run.main(["--portfolio"]) out = capsys.readouterr().out assert rc == 0 # exhaustion is a structured stop, not a failure — it does not fail the pass assert "budget" in out.lower() for number in ("500", "470", "30", "120"): assert number in out, out def test_no_budget_stop_line_when_the_pass_was_not_stopped(stub_portfolio, capsys) -> None: """Blade 10 (control) — blade 9 cannot pass on an unconditional line.""" stub_portfolio(_result()) run.main(["--portfolio"]) assert "budget" not in capsys.readouterr().out.lower()