feat(p16): --max-rounds/--max-tokens -- the cap on a PAID run had no operator door
B2's free drill earned its keep on the first command. STATE.md, docs/2026-09-12-p14-kontekstsett.md and the order all publish the same stress command ending "--max-rounds 8 --max-tokens 120000". Measured: run.py accepts neither, all four --live-dry-run drills refused with "unrecognized arguments", and main() never passed max_rounds/max_tokens to run_project at all -- so every CLI run ever made was silently bound to _DEFAULT_MAX_ROUNDS=3 / _DEFAULT_MAX_TOKENS=100_000, with no way to raise or lower the cap on a run being paid for. Three surfaces described a door that did not exist. Widening, never breaking: both flags default to exactly those values, so every existing invocation is byte-identical. Wired to BOTH dispatches -- run_portfolio takes the same two parameters and main() dropped them there too -- and refused by name in report mode, which returns above every dispatch (the F4 silent-drop gap). Load-bearing MEASURED (6 arms, ALL RED before the fix), four mutations all red, green control 1669/5 (from 1663/5, superset, 0 removed), golden BYTE-UNCHANGED (ea8c534...). MEASURED, REPORTED, NOT FIXED: single-project mode still requires --docs-dir even when --bundle-dir is given and docs_dir is unused on the bundle path, so the documented command would have refused for that reason too. The README's own form works; loosening the guard is its own call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
f21007c858
commit
5f94c92476
4 changed files with 198 additions and 0 deletions
22
CLAUDE.md
22
CLAUDE.md
|
|
@ -2444,6 +2444,28 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
|
||||||
den, ikke at modellen sa hvorfor (det leses for hand og rapporteres merket manuelt); og en
|
den, ikke at modellen sa hvorfor (det leses for hand og rapporteres merket manuelt); og en
|
||||||
helbase-snippet som baerer en `ref` er svakere bevis enn modellens eget `measure` — derfor er de
|
helbase-snippet som baerer en `ref` er svakere bevis enn modellens eget `measure` — derfor er de
|
||||||
to halvdelene skilt i utfallet.
|
to halvdelene skilt i utfallet.
|
||||||
|
- **Taket pa en BETALT kjoring hadde ingen operatorflate, og tre flater beskrev en dor som ikke
|
||||||
|
fantes (P16 B2, 14.09):** `STATE.md`, `docs/2026-09-12-p14-kontekstsett.md § 4.1` og ordre
|
||||||
|
`20260914T091846Z` publiserer alle den samme stresskommandoen, som ender `--max-rounds 8
|
||||||
|
--max-tokens 120000`. MALT: `run.py` tok ingen av dem, alle fire gratis `--live-dry-run` nektet
|
||||||
|
med `unrecognized arguments`, og `main()` sendte **aldri** `max_rounds`/`max_tokens` videre — sa
|
||||||
|
HVER CLI-kjoring noensinne var stille bundet til `_DEFAULT_MAX_ROUNDS=3` /
|
||||||
|
`_DEFAULT_MAX_TOKENS=100_000`, uten mate for en operator a heve eller senke taket pa noe de
|
||||||
|
betalte for. Fase-3-klassen (en pastand flaten gjor om seg selv), spredt over operatorens egne
|
||||||
|
instruksjoner. **UTVIDELSE, aldri bryting:** begge flagg DEFAULTER til nettopp de verdiene, sa
|
||||||
|
hver eksisterende invokasjon er byte-identisk; det som endrer seg er at taket kan UTTALES.
|
||||||
|
**Wiret til BEGGE dispatcher** (`run_project` OG `run_portfolio` — sistnevnte tar de samme to
|
||||||
|
parameterne og `main()` droppet dem der ogsaa, sa et portefoljepass kunne heller ikke kappes), og
|
||||||
|
**nektet i report-modus VED NAVN**: report-modus returnerer OVER hver dispatch, sa en utelatelse
|
||||||
|
er et stille DROPP, ikke en nekt (F4-gapet) — og de er skillbare kun fordi argparse-defaulten ER
|
||||||
|
den historiske verdien. Load-bearing MALT (`tests/test_budget_flags_loadbearing.py`, 6 armer, ALLE
|
||||||
|
RODE for fiksen), fire mutasjoner alle rode + gronn kontroll **1669/5** og golden BYTE-UENDRET:
|
||||||
|
M12 flaggene nar aldri `run_project` (2) · M13 nar aldri `run_portfolio` (1) · M14 droppet fra
|
||||||
|
`report_forbidden` (2) · M15 defaulten drifter fra det kjoringene brukte (3). **Aerlighets-grense,
|
||||||
|
MALT og IKKE fikset:** enkeltprosjekt-modus krever fortsatt `--docs-dir` (`run.py:2953`) selv nar
|
||||||
|
`--bundle-dir` er oppgitt og `docs_dir` er ubrukt pa bundle-stien — den dokumenterte kommandoen
|
||||||
|
ville nektet av DEN grunnen ogsaa; READMEs egen form (`--docs-dir <bundle> --bundle-dir <bundle>`)
|
||||||
|
er det som virker, og a lose opp guarden er en egen operatorbeslutning.
|
||||||
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
||||||
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -629,6 +629,11 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
||||||
--derive-cost-baseline
|
--derive-cost-baseline
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- **Capping a run** — `--max-rounds` / `--max-tokens`. Both default to the values a CLI run was
|
||||||
|
always bound to (3 rounds, 100 000 tokens), so nothing changes unless you state a cap; stating one
|
||||||
|
is how you raise or lower what a paid run may spend. They apply to `--portfolio` too, and are
|
||||||
|
refused by name in `--report` mode, where a cap would bind nothing.
|
||||||
|
|
||||||
- **Judging a finished run against a fasit** — `python -m portfolio_optimiser.stress`. A run's
|
- **Judging a finished run against a fasit** — `python -m portfolio_optimiser.stress`. A run's
|
||||||
outbox already carries the evidence (which documents the debate opened, which files the stamp
|
outbox already carries the evidence (which documents the debate opened, which files the stamp
|
||||||
cites, which approaches validated). This reads it against a context set's own `fasit.json` and
|
cites, which approaches validated). This reads it against a context set's own `fasit.json` and
|
||||||
|
|
|
||||||
|
|
@ -2656,6 +2656,21 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
help="the expert's reasoning behind --decision (required with it; an expert verdict is a "
|
help="the expert's reasoning behind --decision (required with it; an expert verdict is a "
|
||||||
"decision AND its reasoning)",
|
"decision AND its reasoning)",
|
||||||
)
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--max-rounds",
|
||||||
|
type=int,
|
||||||
|
default=_DEFAULT_MAX_ROUNDS,
|
||||||
|
help=(
|
||||||
|
"per-run round cap (P16 B2). DEFAULTS to what every CLI run was implicitly bound to "
|
||||||
|
"before this flag existed, so nothing changes unless it is stated"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--max-tokens",
|
||||||
|
type=int,
|
||||||
|
default=_DEFAULT_MAX_TOKENS,
|
||||||
|
help="per-run token cap (P16 B2). Same default, same reason as --max-rounds",
|
||||||
|
)
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--live-dry-run",
|
"--live-dry-run",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
|
|
@ -2819,6 +2834,11 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
# F4 class exactly: report mode returns above every dispatch, so an omission here is a
|
# F4 class exactly: report mode returns above every dispatch, so an omission here is a
|
||||||
# silent drop and not a refusal.
|
# silent drop and not a refusal.
|
||||||
"--mandate": args.mandate is not None,
|
"--mandate": args.mandate is not None,
|
||||||
|
# P16 B2, same rung: a cap stated in report mode binds nothing, and report mode returns
|
||||||
|
# above the dispatch that would have honoured it. Distinguishable only because the
|
||||||
|
# argparse default IS the historic value, so "stated" and "omitted" differ.
|
||||||
|
"--max-rounds": args.max_rounds != _DEFAULT_MAX_ROUNDS,
|
||||||
|
"--max-tokens": args.max_tokens != _DEFAULT_MAX_TOKENS,
|
||||||
}
|
}
|
||||||
if any(report_forbidden.values()):
|
if any(report_forbidden.values()):
|
||||||
print(
|
print(
|
||||||
|
|
@ -3787,6 +3807,8 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
client_factory=scripted_client_factory,
|
client_factory=scripted_client_factory,
|
||||||
mandate=mandate,
|
mandate=mandate,
|
||||||
mcp_servers=mcp_servers,
|
mcp_servers=mcp_servers,
|
||||||
|
max_rounds=args.max_rounds,
|
||||||
|
max_tokens=args.max_tokens,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
except (ValueError, FileNotFoundError, ValidationError) as exc:
|
except (ValueError, FileNotFoundError, ValidationError) as exc:
|
||||||
|
|
@ -3867,6 +3889,8 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
if args.embedder_config
|
if args.embedder_config
|
||||||
else None
|
else None
|
||||||
),
|
),
|
||||||
|
max_rounds=args.max_rounds,
|
||||||
|
max_tokens=args.max_tokens,
|
||||||
outbox_dir=args.outbox_dir,
|
outbox_dir=args.outbox_dir,
|
||||||
run_id=args.run_id,
|
run_id=args.run_id,
|
||||||
prepass_payload=prepass_payload,
|
prepass_payload=prepass_payload,
|
||||||
|
|
@ -3943,6 +3967,8 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
if args.embedder_config
|
if args.embedder_config
|
||||||
else None
|
else None
|
||||||
),
|
),
|
||||||
|
max_rounds=args.max_rounds,
|
||||||
|
max_tokens=args.max_tokens,
|
||||||
outbox_dir=args.outbox_dir,
|
outbox_dir=args.outbox_dir,
|
||||||
run_id=args.run_id,
|
run_id=args.run_id,
|
||||||
prepass_payload=prepass_payload,
|
prepass_payload=prepass_payload,
|
||||||
|
|
|
||||||
145
tests/test_budget_flags_loadbearing.py
Normal file
145
tests/test_budget_flags_loadbearing.py
Normal file
|
|
@ -0,0 +1,145 @@
|
||||||
|
"""P16 B2 - the CLI had NO door onto a paid run's cap, and the documented command proved it.
|
||||||
|
|
||||||
|
**The measured silence.** ``STATE.md``, ``docs/2026-09-12-p14-kontekstsett.md § 4.1`` and order
|
||||||
|
``20260914T091846Z`` all publish the same stress command, ending ``--max-rounds 8 --max-tokens
|
||||||
|
120000``. Measured 14.09: ``run.py`` accepts neither flag, all four free ``--live-dry-run`` drills
|
||||||
|
refused with ``unrecognized arguments``, and ``main()`` never passed ``max_rounds``/``max_tokens``
|
||||||
|
to ``run_project`` at all - so **every CLI run ever made was silently bound to the defaults**
|
||||||
|
(``max_rounds=3``, ``max_tokens=100_000``) with no way for an operator to raise or lower the cap on
|
||||||
|
a run they were paying for. Three surfaces described a door that did not exist: the Fase-3 class
|
||||||
|
(a claim the surface makes about itself), spread across the operator's own instructions.
|
||||||
|
|
||||||
|
**Widening, never breaking.** Both flags DEFAULT to the values ``run_project`` already used, so
|
||||||
|
every invocation that exists is byte-identical; what changes is that the cap can now be stated.
|
||||||
|
|
||||||
|
**Wired to BOTH dispatches, and refused in report mode.** ``run_portfolio`` takes the same two
|
||||||
|
parameters and ``main()`` dropped them there too, so a portfolio pass could not be capped either.
|
||||||
|
Report mode returns ABOVE every dispatch, so a flag left out of ``report_forbidden`` is a SILENT
|
||||||
|
DROP rather than a refusal - the gap F4 measured on ``--plan-review``.
|
||||||
|
|
||||||
|
Arms: (a) the flags parse and reach ``run_project`` * (b) they reach ``run_portfolio`` *
|
||||||
|
(c) the defaults are unchanged when the flags are absent * (d) report mode refuses each by name *
|
||||||
|
(e) the free dry-run drill accepts the documented command form.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from portfolio_optimiser import run as run_module
|
||||||
|
|
||||||
|
_REPO = Path(__file__).resolve().parents[1]
|
||||||
|
|
||||||
|
|
||||||
|
def _base(root: Path) -> Path:
|
||||||
|
base = root / "b"
|
||||||
|
(base / "krav").mkdir(parents=True)
|
||||||
|
(base / "index.md").write_text("---\nbundle_id: b\n---\n\n- [c](krav/c.md)\n", encoding="utf-8")
|
||||||
|
(base / "krav" / "c.md").write_text(
|
||||||
|
'---\ntype: concept\ntitle: "C"\n---\n\nbody\n', encoding="utf-8"
|
||||||
|
)
|
||||||
|
return base
|
||||||
|
|
||||||
|
|
||||||
|
def _capture(monkeypatch: pytest.MonkeyPatch, target: str) -> dict[str, Any]:
|
||||||
|
"""Record the kwargs main() hands the named coroutine, then stop the run."""
|
||||||
|
seen: dict[str, Any] = {}
|
||||||
|
|
||||||
|
async def _fake(*args: Any, **kwargs: Any) -> Any:
|
||||||
|
seen.update(kwargs)
|
||||||
|
raise SystemExit(0)
|
||||||
|
|
||||||
|
monkeypatch.setattr(run_module, target, _fake)
|
||||||
|
return seen
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_the_two_flags_reach_run_project(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
|
||||||
|
seen = _capture(monkeypatch, "run_project")
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
b = str(_base(tmp_path))
|
||||||
|
run_module.main(
|
||||||
|
[
|
||||||
|
"P1",
|
||||||
|
"--profile",
|
||||||
|
"local",
|
||||||
|
"--docs-dir",
|
||||||
|
b,
|
||||||
|
"--bundle-dir",
|
||||||
|
b,
|
||||||
|
"--max-rounds",
|
||||||
|
"8",
|
||||||
|
"--max-tokens",
|
||||||
|
"120000",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
assert seen["max_rounds"] == 8
|
||||||
|
assert seen["max_tokens"] == 120000
|
||||||
|
|
||||||
|
|
||||||
|
def test_b_the_two_flags_reach_run_portfolio(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
seen = _capture(monkeypatch, "run_portfolio")
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
run_module.main(
|
||||||
|
["--portfolio", "--profile", "local", "--max-rounds", "9", "--max-tokens", "77000"]
|
||||||
|
)
|
||||||
|
assert seen["max_rounds"] == 9
|
||||||
|
assert seen["max_tokens"] == 77000
|
||||||
|
|
||||||
|
|
||||||
|
def test_c_absent_flags_keep_the_values_every_run_so_far_used(
|
||||||
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""Widening, never breaking: the defaults ARE what main() bound implicitly before."""
|
||||||
|
seen = _capture(monkeypatch, "run_project")
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
b = str(_base(tmp_path))
|
||||||
|
run_module.main(["P1", "--profile", "local", "--docs-dir", b, "--bundle-dir", b])
|
||||||
|
assert seen["max_rounds"] == run_module._DEFAULT_MAX_ROUNDS == 3
|
||||||
|
assert seen["max_tokens"] == run_module._DEFAULT_MAX_TOKENS == 100_000
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("flag,value", [("--max-rounds", "8"), ("--max-tokens", "120000")])
|
||||||
|
def test_d_report_mode_refuses_each_flag_by_name(
|
||||||
|
flag: str, value: str, tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||||
|
) -> None:
|
||||||
|
ledger = tmp_path / "l.json"
|
||||||
|
ledger.write_text("[]", encoding="utf-8")
|
||||||
|
# The control: the SAME argv without the flag is ACCEPTED, so rc 1 below is the mutant's
|
||||||
|
# opposite outcome and not the fixture refusing for a reason of its own.
|
||||||
|
assert run_module.main(["--report", "--ledger", str(ledger)]) == 0
|
||||||
|
assert run_module.main(["--report", "--ledger", str(ledger), flag, value]) == 1
|
||||||
|
assert "mode-exclusive" in capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
|
def test_e_the_documented_dry_run_command_form_is_accepted(tmp_path: Path) -> None:
|
||||||
|
"""The B2 drill itself: the command STATE.md and the order publish must at least parse."""
|
||||||
|
base = _base(tmp_path)
|
||||||
|
proc = subprocess.run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
"-m",
|
||||||
|
"portfolio_optimiser.run",
|
||||||
|
"P1",
|
||||||
|
"--profile",
|
||||||
|
"local",
|
||||||
|
"--docs-dir",
|
||||||
|
str(base),
|
||||||
|
"--bundle-dir",
|
||||||
|
str(base),
|
||||||
|
"--max-rounds",
|
||||||
|
"8",
|
||||||
|
"--max-tokens",
|
||||||
|
"120000",
|
||||||
|
"--live-dry-run",
|
||||||
|
],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
cwd=_REPO,
|
||||||
|
)
|
||||||
|
assert "unrecognized arguments" not in proc.stderr, proc.stderr
|
||||||
|
assert proc.returncode == 0, proc.stderr
|
||||||
Loading…
Add table
Add a link
Reference in a new issue