fix(p20): code_forms follows ITS OWN approach, and the announcement has a witness [skip-docs]
Two defects the mutation battery and the paid round found, both measured before being touched. (1) code_forms described the WRONG candidate. Every per-approach artefact copied the run's stamp and overrode only validator_decision, so an artefact about approach 2 reported approach 1's codes. Measured in BOTH round 3 and round 4 -- and stress.py, which reads this field before re-deriving, then produced an EMPTY prose_codes for every approach but the first, which is what round 3's table was built on. The field's own comment already says it is stamped "off the proposal being stamped"; run-level was the drift, not the intent. Model, citations and token usage stay the run's, because they are the run's. (2) The C2 announcement seam had no witness. Mutation C-iii reverted the call site to `args.project_id or "the portfolio"` and the WHOLE suite stayed green (1808/5): all three arms drove announced_subject directly. The missing arm drives main() on a free dry run and reads the announcement off STDOUT, where an operator reads it, and is red against exactly that mutation. Sixteen mutations, ALL red against the whole suite. Green control 1809/5 (from 1781, +28, 0 removed), golden demo-transcript.stdout BYTE-UNCHANGED (shasum -a 1 of the CONTENT = ea8c534773acdbe41ae68f2c55724d69aaf8be4f). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
c8f0c8f7c4
commit
e513bc97ad
4 changed files with 139 additions and 3 deletions
11
CLAUDE.md
11
CLAUDE.md
|
|
@ -2805,7 +2805,12 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
|
|||
ville endret `total_matches` for hvert navigatørkall. `code_forms` får en TREDJE verdi,
|
||||
`requirement`, for en kode basen FAKTISK erklærer (ordrens kjent-negative (c): alle
|
||||
`must_cite`-referanser klassifiseres slik) — rapport, aldri gaten, som fyrer på komplementet.
|
||||
Load-bearing MÅLT (`tests/test_requirement_number_gate_loadbearing.py`, 11 armer).
|
||||
Load-bearing MÅLT (`tests/test_requirement_number_gate_loadbearing.py`, 12 armer), ni mutasjoner
|
||||
alle røde mot HELE suiten. **FUNN UNDERVEIS, FIKSET:** `code_forms` bar den SELEKTERTE kandidatens
|
||||
koder i HVERT per-approach-artefakt (målt i BÅDE runde 3 og runde 4) — `stamp.model_copy` overstyrte
|
||||
kun `validator_decision`, mens feltets eget kommentar sier det er stemplet «off the proposal being
|
||||
stamped». Følgen var at dommeren, som leser feltet FØRST, rapporterte TOM `prose_codes` for hver
|
||||
approach unntatt den første.
|
||||
- **Erklæringen svarer med DOKUMENTETS egne ord, og kommisjonens kriterier når leseren som kan
|
||||
handle på dem (P20 DEL A, 15.09):** P19 DEL A gjorde at en retning MÅ navngi kravet som binder
|
||||
den, og rungen virker — MÅLT ga runde 3 ni erklæringer over fem betalte kjøringer og P17b fire
|
||||
|
|
@ -2825,7 +2830,9 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
|
|||
hva som holder hver prompt i hver ukommisjonert kjøring byte-identisk, demoens inkludert.
|
||||
**A2s plassering er MÅLT:** på utforskningsstien finnes ingenting å bære — `main()` sender
|
||||
`explore()` ingen `success_criteria` i det hele tatt, så objektivet ER prompten og når oppgaven
|
||||
alt. Load-bearing MÅLT (`tests/test_right_requirement_loadbearing.py`, 8 armer).
|
||||
alt. Load-bearing MÅLT (`tests/test_right_requirement_loadbearing.py`, 8 armer), fire mutasjoner røde.
|
||||
**ORDRENS SPÅDDE SIGNATUR BLE FALSIFISERT:** A-iii skulle gjøre goldenen rød, men demoen kjører uten
|
||||
mandat, så `criteria` er tom uansett og bare rendererens egen arm faller.
|
||||
- **En parse-feil brenner ikke lenger rundeboka i stillhet, og annonseringen navngir hva den
|
||||
handler om (P20 DEL C, 15.09):** `_fetch_parsed` prøvde på nytt med den BYTE-IDENTISKE prompten.
|
||||
MÅLT (P19 F4): `kontrakt-sorasen-04` etterlot `{run_id}-parse-failures.json` med **elleve** rader,
|
||||
|
|
|
|||
|
|
@ -1740,7 +1740,20 @@ async def run_project(
|
|||
"validated"
|
||||
if isinstance(approach_outcome, ValidatedProposal)
|
||||
else "rejected"
|
||||
)
|
||||
),
|
||||
# P20: ``code_forms`` must follow ITS OWN approach too, for
|
||||
# ``validator_decision``'s reason exactly. MEASURED on round 3 and
|
||||
# round 4: every per-approach artefact carried the SELECTED proposal's
|
||||
# codes, so ``code_forms`` described a candidate the artefact is not
|
||||
# about — and the judge, which reads this field first, reported an
|
||||
# empty ``prose_codes`` for every approach but one. The field's own
|
||||
# docstring already says it is stamped "off the proposal being
|
||||
# stamped"; run-level was the drift, not the intent. Model, citations
|
||||
# and token usage stay the RUN's, because they are the run's.
|
||||
"code_forms": classify_codes(
|
||||
[i.code for i in approach_outcome.proposal.affected_items],
|
||||
delivered,
|
||||
),
|
||||
}
|
||||
),
|
||||
checker_verdict=checker_decision,
|
||||
|
|
|
|||
|
|
@ -28,9 +28,12 @@ What each arm pins:
|
|||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser.budget import Budget, TokenMeter
|
||||
from portfolio_optimiser.generate import ParseFailure, _build_messages, generate_via_llm
|
||||
from portfolio_optimiser.reference_domain import Project
|
||||
|
|
@ -122,6 +125,42 @@ def test_an_unresolvable_base_falls_back_to_its_directory_name(tmp_path: Path) -
|
|||
assert announced_subject(None, (str(missing),)) == "not-a-base"
|
||||
|
||||
|
||||
def test_the_cli_prints_the_routed_bases(
|
||||
tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
"""(e2) The WIRING, not the renderer.
|
||||
|
||||
MEASURED: mutation C-iii reverted the call site to ``args.project_id or "the portfolio"`` and
|
||||
the WHOLE suite stayed green (1808/5) — the renderer had three arms and the call site none, so
|
||||
the seam C2 exists for was unwitnessed. This arm drives ``main()`` on a free dry run and reads
|
||||
the announcement off stdout, which is where an operator reads it.
|
||||
"""
|
||||
from portfolio_optimiser.run import main
|
||||
|
||||
mandate = tmp_path / "m.json"
|
||||
mandate.write_text(
|
||||
json.dumps({"objective": "cut cost", "approaches": [], "allow_own_proposals": True}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
rc = main(
|
||||
[
|
||||
"--across-bundle",
|
||||
str(_TUNNEL),
|
||||
"--mandate",
|
||||
str(mandate),
|
||||
"--run-id",
|
||||
"r1",
|
||||
"--outbox-dir",
|
||||
str(tmp_path / "out"),
|
||||
"--live-dry-run",
|
||||
]
|
||||
)
|
||||
out = capsys.readouterr().out
|
||||
assert rc == 0, out
|
||||
assert "Run mandate for tunnel-hauglia" in out
|
||||
assert "the portfolio" not in out
|
||||
|
||||
|
||||
def test_the_two_older_subjects_are_unchanged() -> None:
|
||||
"""(g) A named project wins; no bases at all is still the portfolio."""
|
||||
assert announced_subject("proj-1", ()) == "proj-1"
|
||||
|
|
|
|||
|
|
@ -273,3 +273,80 @@ def test_a_run_composes_the_vocabulary_from_the_base_it_opened(tmp_path: Path) -
|
|||
)
|
||||
assert result.provenance.code_forms == {"12.1": "requirement"}
|
||||
assert result.provenance.validator_decision == "validated"
|
||||
|
||||
|
||||
def test_each_approachs_artefact_carries_its_own_code_forms(tmp_path: Path) -> None:
|
||||
"""(j) MEASURED on round 3 AND round 4: every per-approach artefact carried the SELECTED
|
||||
proposal's codes.
|
||||
|
||||
``code_forms``' own comment says it is derived "off the proposal being stamped", and the
|
||||
per-approach branch copied the run's stamp while overriding only ``validator_decision``. So an
|
||||
artefact about approach 2 reported approach 1's codes — and ``stress.py``, which reads this
|
||||
field before re-deriving, then produced an EMPTY ``prose_codes`` for every approach but the
|
||||
first, because none of that approach's codes was a key in the map it was handed.
|
||||
|
||||
Run-level was the drift, not the intent: model, citations and token usage ARE the run's, and
|
||||
they stay so.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
from agent_framework import BaseChatClient
|
||||
|
||||
from portfolio_optimiser.mandate import Approach, Mandate
|
||||
from portfolio_optimiser.run import run_project
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
|
||||
base = tmp_path / "mini"
|
||||
base.mkdir()
|
||||
(base / "index.md").write_text(
|
||||
"---\nbundle_id: mini\n---\n\n- [Krav](krav.md) — one clause.\n", encoding="utf-8"
|
||||
)
|
||||
(base / "krav.md").write_text(
|
||||
"---\ntype: Krav\ntitle: Krav 4.1.2-1\nprosessnr: '12.1'\n---\n\nLines 12.1 and PRIS-EN.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(base / "validator-input.json").write_text(
|
||||
json.dumps({"project_id": "mini-p", "measure": "m", "affected_codes": ["12.1"]}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
outbox = tmp_path / "out"
|
||||
|
||||
# Each approach's prompt names its own label, so the selector answers with ITS OWN code.
|
||||
def select(prompt: str, role: str) -> str:
|
||||
if role == "checker":
|
||||
return "Reasoning holds.\nVERDICT: APPROVE"
|
||||
code = "PRIS-EN" if "SECOND" in prompt else "12.1"
|
||||
return (
|
||||
'{"measure":"m","affected_items":[{"code":"%s","quantity":10,"unit_cost":1000}],'
|
||||
'"claimed_saving_nok":1000}' % code
|
||||
)
|
||||
|
||||
def factory(role: str) -> BaseChatClient:
|
||||
return ScriptedChatClient(reply_selector=select, role=role)
|
||||
|
||||
asyncio.run(
|
||||
run_project(
|
||||
"mini-p",
|
||||
"local",
|
||||
docs_dir=str(base),
|
||||
bundle_dir=str(base),
|
||||
client_factory=factory,
|
||||
mandate=Mandate(
|
||||
objective="o",
|
||||
approaches=(
|
||||
Approach(id="a1", label="FIRST", description="d"),
|
||||
Approach(id="a2", label="SECOND", description="d"),
|
||||
),
|
||||
allow_own_proposals=False,
|
||||
),
|
||||
outbox_dir=str(outbox),
|
||||
run_id="r1",
|
||||
)
|
||||
)
|
||||
forms = {}
|
||||
for name in ("a1", "a2"):
|
||||
payload = json.loads((outbox / f"r1-{name}-proposal.json").read_text(encoding="utf-8"))
|
||||
codes = [i["code"] for i in payload["proposal"]["affected_items"]]
|
||||
forms[name] = (codes, payload["provenance"]["code_forms"])
|
||||
assert forms["a1"] == (["12.1"], {"12.1": "requirement"}), forms["a1"]
|
||||
assert forms["a2"] == (["PRIS-EN"], {"PRIS-EN": "identifier"}), forms["a2"]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue