F3 and F4, the two findings the S7 acid test (session 98) reported and left. The order required both descriptions to be treated as PREMISES. One held; the other was felled before anything was built on it. F3 -- premise FELLED, asymmetry real. The order read arm C's two refused calls as "the path names a document that EXISTS". Measured against the base that ran: its root holds 27 directories named del-ii-bilag-N-... and 12 documents named inbox-del-ii-bilag-N-....md, and the requested path matches NEITHER -- it is the directory naming convention applied to a document whose real name carries an inbox- prefix. So the two live rounds were the UNKNOWN-path class, and this delivery does NOT recover them (gated). What IS real: read_file on a directory has named read_dir since session 95, while read_dir on a document named neither the rung nor the path. okf.DocumentPathRefused closes that one direction -- a ValueError, a SIBLING of BundlePathNotFound rather than a subclass, built from context_files (never files) and through the same in_dimension predicate the listing uses, quoting the document's REAL name so what it hands back resolves. F4 -- premise HELD, option (c) felled by measurement. All four live artefacts stamped cost_baseline_anchored: False and each arm invented its cost codes. derive_cost_baseline refuses against the delivered base: K2's price schedule is a pandoc SIMPLE table with ONE column header, so making --derive-cost-baseline reachable there would mean inventing a rule for an unmeasured form -- MAJOR-4's own honesty limit. Chose (b) over (a): --require-cost-baseline / run_project(require_cost_baseline=...), OPT-IN and never default, so every bundle without a cost-baseline.json runs unchanged. The gate sits where both branches have bound baseline and ABOVE the dry-run cut, so it fires on the free trip too and, on the paid one, before the first model call. Three CLI refusals by name, each with an rc-0 control. 12 mutations, all red against the WHOLE suite. Green control 1493/5 -> 1511/5 (+18 node-ids, 0 removed); golden demo-transcript.stdout BYTE-UNCHANGED (shasum -a 1 of the CONTENT = ea8c534773acdbe41ae68f2c55724d69aaf8be4f). No paid run: both findings measured offline. Measurement: docs/2026-09-08-f3-f4-nekten-og-forankringen.md Order: 20260908T020419Z-5837110336-from-portfolio-optimiser Co-Authored-By: Claude <Opus 5>
240 lines
10 KiB
Python
240 lines
10 KiB
Python
"""F4 - a run that must be anchored can say so, and is refused before it costs anything.
|
|
|
|
**The measurement** (``docs/2026-09-07-syretest-s7-prepass-k2.md`` § 8, re-measured here against
|
|
the four artefacts the live run left). All three paid arms stamped
|
|
``cost_baseline_anchored: False``, so the validator's stage 0 - the ONE stage that tells a
|
|
fabricated cost line from a real one - was skipped, and every arm invented its codes:
|
|
``ENGRAVE_MARK``, ``RITB-HOURS``/``SYSINT-HOURS``, ``Material_Cost_Concrete``. Stages 2/4/5 judged
|
|
those numbers against each other and did their job; nothing tied a single line to K2.
|
|
|
|
**Option (c) is felled by measurement, not by preference.** The order offers "make
|
|
``--derive-cost-baseline`` reachable where it is not today". Measured against the base that ran:
|
|
``derive_cost_baseline`` refuses with *no concept file carries a markdown table whose header names
|
|
all three of code/quantity/unit_cost* - K2's delivered price schedule is a pandoc SIMPLE table with
|
|
ONE column header (``Prisskjema``) and every value collapsed into it. Reaching it would mean
|
|
inventing a rule for a form nobody has measured, which is precisely the honesty limit MAJOR-4
|
|
wrote down for itself.
|
|
|
|
**Option (b), and why not (a).** The visibility exists (``cost_baseline_notice``) and is honest;
|
|
what it cannot do is stop a machine-readable artefact that says ``validator_decision: validated``
|
|
over lines nothing anchored. So: an OPT-IN flag, never a default. Every bundle without a
|
|
``cost-baseline.json`` - every commons-owned golden - runs exactly as before, which is what (a)
|
|
was protecting; a caller who needs the guarantee asks for it by name.
|
|
|
|
The gate sits at the ONE place both branches have bound ``baseline`` and BEFORE the dry-run cut, so
|
|
it fires on the free trip too (arm (d)) and, on the paid one, before the first model call (arm (a)
|
|
asserts NULL calls, never merely the exception - at the exit code a refusal after the spend looks
|
|
identical to one before).
|
|
|
|
Arms: (a) refused, zero model calls * (b) control: without the flag the same base runs *
|
|
(c) an anchored base is untouched * (d) the dry run refuses too, with its own control *
|
|
(e) it composes with ``--derive-cost-baseline`` * (f)(g)(h) three CLI refusals BY NAME, each with
|
|
an rc-0 control on an argv that would otherwise be accepted * (i) the CLI wiring, measured on
|
|
calls * (j) the channel.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import shutil
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
from portfolio_optimiser import run
|
|
from portfolio_optimiser.simulation import ScriptedChatClient
|
|
|
|
_FIXTURES = Path(__file__).parent / "fixtures"
|
|
_PRICED = str(_FIXTURES / "k2-prisskjema-SYNTETISK")
|
|
_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
|
|
#: A base that SHIPS a hand-written ``cost-baseline.json`` - the anchored control.
|
|
_ANCHORED_SOURCE = _EXAMPLES / "tunnel-hauglia"
|
|
|
|
_IR_PROJECTION = {
|
|
"project_id": "K2",
|
|
"measure": "PLACEHOLDER - authored by this test to satisfy the bundle contract",
|
|
"affected_items": [{"code": "21.1", "quantity": 1250, "unit_cost": 850.0}],
|
|
"claimed_saving_nok": 1000.0,
|
|
}
|
|
|
|
|
|
def _runnable(source: str, tmp_path: Path, *, name: str = "runnable") -> str:
|
|
root = tmp_path / name
|
|
shutil.copytree(source, root)
|
|
(root / "validator-input.json").write_text(json.dumps(_IR_PROJECTION), encoding="utf-8")
|
|
return str(root)
|
|
|
|
|
|
def _scripted(sink: list[str] | None = None) -> Any:
|
|
return lambda role: ScriptedChatClient(sink=sink, role=role, default_reply="ok")
|
|
|
|
|
|
# --- (a)/(b)/(c) the library seam -----------------------------------------------------------------
|
|
|
|
|
|
async def test_an_unanchored_run_is_refused_before_it_costs_anything(tmp_path: Path) -> None:
|
|
"""(a) The refusal, and the assert that makes it worth having: NULL model calls. Measured on
|
|
calls rather than on the exception, because a refusal placed after the spend raises the same
|
|
exception (session 57's rule, session 82's arm)."""
|
|
bundle_dir = _runnable(_PRICED, tmp_path)
|
|
sink: list[str] = []
|
|
|
|
with pytest.raises(run.UnanchoredRunRefused):
|
|
await run.run_project(
|
|
"K2",
|
|
"local",
|
|
docs_dir=bundle_dir,
|
|
bundle_dir=bundle_dir,
|
|
require_cost_baseline=True,
|
|
client_factory=_scripted(sink),
|
|
)
|
|
|
|
assert sink == [], f"the run reached the model before it was refused ({len(sink)} calls)"
|
|
|
|
|
|
async def test_control_without_the_flag_the_same_base_runs(tmp_path: Path) -> None:
|
|
"""(b) The default is byte-for-byte the path every existing caller takes - which is what
|
|
option (a) of the order was protecting, kept rather than argued away."""
|
|
bundle_dir = _runnable(_PRICED, tmp_path)
|
|
|
|
report = await run.run_project(
|
|
"K2", "local", docs_dir=bundle_dir, bundle_dir=bundle_dir, live_dry_run=True
|
|
)
|
|
|
|
assert isinstance(report, run.DryRunReport)
|
|
assert report.cost_baseline_anchored is False
|
|
|
|
|
|
async def test_an_anchored_base_is_untouched_by_the_flag(tmp_path: Path) -> None:
|
|
"""(c) The gate reads the SAME ``baseline`` the validator is handed, so a base that ships the
|
|
file passes it. Without this arm the flag could be a refusal that always fires."""
|
|
bundle_dir = _runnable(str(_ANCHORED_SOURCE), tmp_path, name="anchored")
|
|
|
|
report = await run.run_project(
|
|
"K2",
|
|
"local",
|
|
docs_dir=bundle_dir,
|
|
bundle_dir=bundle_dir,
|
|
require_cost_baseline=True,
|
|
live_dry_run=True,
|
|
)
|
|
|
|
assert isinstance(report, run.DryRunReport)
|
|
assert report.cost_baseline_anchored is True
|
|
|
|
|
|
# --- (d) the free trip refuses too ----------------------------------------------------------------
|
|
|
|
|
|
async def test_the_dry_run_refuses_rather_than_reporting_an_unanchored_run(tmp_path: Path) -> None:
|
|
"""(d) The gate is ABOVE the dry-run cut on purpose: the dry run's whole job is to say what a
|
|
real run would do, and it is the cheapest place to learn that this one may not run at all."""
|
|
bundle_dir = _runnable(_PRICED, tmp_path)
|
|
|
|
with pytest.raises(run.UnanchoredRunRefused):
|
|
await run.run_project(
|
|
"K2",
|
|
"local",
|
|
docs_dir=bundle_dir,
|
|
bundle_dir=bundle_dir,
|
|
require_cost_baseline=True,
|
|
live_dry_run=True,
|
|
)
|
|
|
|
|
|
# --- (e) it composes with the derivation ----------------------------------------------------------
|
|
|
|
|
|
async def test_it_composes_with_the_derivation(tmp_path: Path) -> None:
|
|
"""(e) ``--derive-cost-baseline`` is the OTHER half of the answer: derive, and the requirement
|
|
is met. Two flags that could not be combined would leave the guarantee unreachable on exactly
|
|
the bases MAJOR-4 was built for."""
|
|
bundle_dir = _runnable(_PRICED, tmp_path)
|
|
|
|
report = await run.run_project(
|
|
"K2",
|
|
"local",
|
|
docs_dir=bundle_dir,
|
|
bundle_dir=bundle_dir,
|
|
derive_cost_baseline=True,
|
|
require_cost_baseline=True,
|
|
live_dry_run=True,
|
|
)
|
|
|
|
assert isinstance(report, run.DryRunReport)
|
|
assert report.cost_baseline_anchored is True
|
|
|
|
|
|
# --- (f)/(g)/(h) three CLI refusals, each with an rc-0 control -------------------------------------
|
|
|
|
|
|
def test_cli_requires_bundle_dir(capsys: pytest.CaptureFixture[str]) -> None:
|
|
"""(f) The road path is anchored by construction, so on it the flag could never fire. A flag
|
|
that cannot fire is a claim the surface makes about itself (the Fase-3 class)."""
|
|
rc = run.main(["P1", "--docs-dir", "docs", "--require-cost-baseline"])
|
|
|
|
assert rc == 1
|
|
assert "--require-cost-baseline" in capsys.readouterr().err
|
|
|
|
|
|
def test_cli_is_refused_in_portfolio_mode(capsys: pytest.CaptureFixture[str]) -> None:
|
|
"""(g) BY NAME, never by falling through to the ``--bundle-dir`` requirement: that message
|
|
would tell an operator who wrote ``--portfolio --require-cost-baseline`` to add the one flag
|
|
this mode also refuses (the ``--derive-cost-baseline`` precedent, and session 57's)."""
|
|
rc = run.main(["--portfolio", "--require-cost-baseline"])
|
|
|
|
assert rc == 1
|
|
err = capsys.readouterr().err
|
|
assert "--portfolio" in err, "the refusal never names the mode that refused it"
|
|
|
|
|
|
def test_cli_is_refused_in_report_mode(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
|
|
"""(h) Report mode returns ABOVE every run dispatch, so an omission from this allowlist is a
|
|
SILENT DROP rather than a refusal - the gap F4 itself measured on ``--plan-review``."""
|
|
ledger = tmp_path / "ledger.json"
|
|
ledger.write_text("[]", encoding="utf-8")
|
|
|
|
assert run.main(["--report", "--ledger", str(ledger)]) == 0, "the control argv must be ACCEPTED"
|
|
capsys.readouterr()
|
|
|
|
rc = run.main(["--report", "--ledger", str(ledger), "--require-cost-baseline"])
|
|
|
|
assert rc == 1
|
|
# The allowlist's own wording, not the flag name: this partition refuses generically, and the
|
|
# discriminator is the rc-0 control above -- the same argv without the flag is ACCEPTED.
|
|
assert "mode-exclusive" in capsys.readouterr().err
|
|
|
|
|
|
# --- (i) the CLI wiring ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_cli_wiring_refuses_the_unanchored_run_without_paying(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
|
|
) -> None:
|
|
"""(i) The flag must REACH ``run_project``. rc 1 alone would also be a parse error, so the
|
|
discriminator is that no model call happened - and the control proves the same argv without
|
|
the flag is accepted and DOES call."""
|
|
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
|
|
bundle_dir = _runnable(_PRICED, tmp_path)
|
|
sink: list[str] = []
|
|
monkeypatch.setattr("portfolio_optimiser.run._default_factory", lambda profile: _scripted(sink))
|
|
|
|
argv = ["K2", "--docs-dir", bundle_dir, "--bundle-dir", bundle_dir]
|
|
assert run.main([*argv, "--live-dry-run"]) == 0, "the control argv must be ACCEPTED"
|
|
|
|
rc = run.main([*argv, "--require-cost-baseline"])
|
|
|
|
assert rc == 1
|
|
assert sink == [], f"the run was refused only after paying for it ({len(sink)} calls)"
|
|
assert "run refused" in capsys.readouterr().err
|
|
|
|
|
|
# --- (j) the channel ------------------------------------------------------------------------------
|
|
|
|
|
|
def test_the_refusal_lands_on_the_refusal_tuple() -> None:
|
|
"""(j) A ``ValueError``, the ``BundleIdMismatch``/``CostBaselineDerivationError`` precedent: an
|
|
argv that is wrong about what this base can offer belongs on the CLI's refusal tuple, never on
|
|
the crash channel."""
|
|
assert issubclass(run.UnanchoredRunRefused, ValueError)
|