feat(p20): the requirement that is RIGHT, and a clause number that is not a price
Three seams, one commit: A, B and C touch the same four modules (run.py carries
the debate task, the grounding composition and the announcement; okf.py carries
one reference-number vocabulary read by both A and B), so splitting them into
three commits would have meant hunk-level staging of entangled files. Stated
rather than silently restructured.
A — the declaration answers with the DOCUMENT's own words. Measured: 13
declarations over round 3 and P17b, not one naming a fasit concept, while the
tool answered {"declared": true, ...} by echoing the caller's own arguments. It
now returns the document's title and req_number, read off Bundle.context_files
(so the type: verdict layer can never be named back), plus the sentence saying
what the declaration binds. A path the base carries as no concept answers with
empty strings rather than refusing. The commission's success_criteria now reach
the DEBATE task through mandate.criteria_block, the one renderer, empty when
there are none — which is what keeps every un-commissioned prompt, and the
golden, byte-identical.
B — a clause number is not a price. THE ORDER'S OWN RULE WAS FELLED BY
MEASUREMENT: it asks to refuse a code that IS declared req_number/prosessnr,
and neither of its two known positives is. n500 declares seksjon 10.4.1..10.4.4
but never the bare 10.4; r761 declares 2727 prosessnr and 2753 seksjon, none of
them 1.10.4, which occurs once, as prose ("iht. vegnormal N200 kap. 1.10.4").
The COMPLEMENT fires on both and closes the hole _ground_against_input already
admits in writing -- "it fails OPEN on a coincidental match". Unanchored run +
requirement-shaped code + the base declares a vocabulary + the code is not in
it -> refused, naming the denominator. All five of kontrakt-sorasen's real
process codes ARE declared and pass, which is what keeps the one context set
built on real codes measurable. Replayed over all 24 codes of round 3 + P17b:
exactly the two known positives flip validated -> rejected, 22 unchanged.
C — a parse failure no longer burns the round ledger blind. _fetch_parsed takes
a BUILDER instead of a finished message list, so the retry carries the parse
reason; measured, kontrakt-sorasen-04 spent 11 of 12 rounds re-asking the same
question. And announced_subject names the routed bases instead of saying "the
portfolio" for a two-base commission.
Suite 1807/5 (from 1781, +26, 0 removed), golden demo-transcript.stdout
BYTE-UNCHANGED (shasum -a 1 of the CONTENT = ea8c534773acdbe41ae68f2c55724d69aaf8be4f),
ruff and mypy clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
64723c5d89
commit
c8f0c8f7c4
12 changed files with 1098 additions and 37 deletions
|
|
@ -121,12 +121,17 @@ def test_the_correction_is_to_read_it_and_then_it_is_accepted() -> None:
|
|||
answer = tools["declare_requirement"].func(
|
||||
bundle_id="tunnel-hauglia", path=path, ref="Krav 12.1"
|
||||
)
|
||||
assert answer == {
|
||||
"declared": True,
|
||||
"bundle_id": "tunnel-hauglia",
|
||||
"path": path,
|
||||
"ref": "Krav 12.1",
|
||||
}
|
||||
# P20/A1 widened the reply: the three arguments PLUS the document's own title and number and
|
||||
# the sentence saying what the declaration binds. Asserted key by key rather than by equality,
|
||||
# because an exact-dict assert here would fail on every future field while saying nothing about
|
||||
# the one property this arm exists for — that the declaration was ACCEPTED and RECORDED.
|
||||
assert answer["declared"] is True
|
||||
assert (answer["bundle_id"], answer["path"], answer["ref"]) == (
|
||||
"tunnel-hauglia",
|
||||
path,
|
||||
"Krav 12.1",
|
||||
)
|
||||
assert set(answer) == {"declared", "bundle_id", "path", "ref", "title", "req_number", "binds"}
|
||||
assert declared == [DeclaredRequirement(bundle_id="tunnel-hauglia", path=path, ref="Krav 12.1")]
|
||||
|
||||
|
||||
|
|
|
|||
129
tests/test_parse_error_feedback_loadbearing.py
Normal file
129
tests/test_parse_error_feedback_loadbearing.py
Normal file
|
|
@ -0,0 +1,129 @@
|
|||
"""P20 DEL C — a parse failure that does not burn the round ledger, and an announcement that
|
||||
names what it is about.
|
||||
|
||||
**C1, what was measured (P19 F4).** ``_fetch_parsed`` retried a malformed reply with the
|
||||
BYTE-IDENTICAL prompt. One round-3 run, ``kontrakt-sorasen-04``, left a
|
||||
``{run_id}-parse-failures.json`` with ELEVEN rows, every one of them the same failure
|
||||
(``claimed_saving_nok`` ≤ 0) — eleven of the run's twelve rounds, spent re-asking a question the
|
||||
model had already answered the same wrong way, because nothing ever told it what was wrong. Step
|
||||
5's ``prior_rejection`` carries VALIDATOR rejections; a reply that never parsed never reaches a
|
||||
validator, so no existing block could carry it.
|
||||
|
||||
**C2, what was measured (P17b F5).** ``--across-bundle`` takes no ``--project-id``, so the
|
||||
announcement — the one thing printed before the first paid call — said "Run mandate for the
|
||||
portfolio" for a commission dispatched across two named knowledge bases.
|
||||
|
||||
What each arm pins:
|
||||
|
||||
(a) the reason reaches the NEXT attempt's prompt, verbatim, asserted on what the client received;
|
||||
(b) attempt 1 is byte-identical: a run whose first reply parses sends the pre-P20 prompt;
|
||||
(c) the block is per-RETRY — once a reply parses, the next attempt does not carry a stale reason;
|
||||
(d) the evidence artefact is unchanged: the verbatim text is still captured (funn 1 stands);
|
||||
(e) the announcement names the routed bases by their DECLARED ids;
|
||||
(f) an unresolvable base falls back to its directory name rather than refusing — the
|
||||
``dimension_label`` precedent, so announcing never changes which error an operator sees;
|
||||
(g) a single-project run and a portfolio pass announce exactly as before.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from portfolio_optimiser.budget import Budget, TokenMeter
|
||||
from portfolio_optimiser.generate import ParseFailure, _build_messages, generate_via_llm
|
||||
from portfolio_optimiser.reference_domain import Project
|
||||
from portfolio_optimiser.run import announced_subject
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
|
||||
_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
|
||||
_TUNNEL = _EXAMPLES / "tunnel-hauglia"
|
||||
|
||||
#: The exact failure ``kontrakt-sorasen-04`` produced eleven times: a well-formed JSON object
|
||||
#: whose ``claimed_saving_nok`` is 0, refused by pydantic before any validator sees it.
|
||||
_UNPARSEABLE = (
|
||||
'{"measure":"m","affected_items":[{"code":"C-1","quantity":1000,"unit_cost":100}],'
|
||||
'"claimed_saving_nok":0}'
|
||||
)
|
||||
#: The grounding this loop declares: P7 requires the code to occur in the input verbatim.
|
||||
_CONTEXT = "The project price schedule carries cost line C-1."
|
||||
_VALID = (
|
||||
'{"measure":"m","affected_items":[{"code":"C-1","quantity":1000,"unit_cost":100}],'
|
||||
'"claimed_saving_nok":5000}'
|
||||
)
|
||||
|
||||
|
||||
def _project() -> Project:
|
||||
return Project(id="p", name="n", description="d", currency="NOK", cost_items=(), docs_dir=".")
|
||||
|
||||
|
||||
def _generate(replies: list[str]) -> tuple[Any, list[str], list[ParseFailure]]:
|
||||
"""Drive the real loop with the repo's ONE scripted client, recording every prompt sent."""
|
||||
sink: list[str] = []
|
||||
failures: list[ParseFailure] = []
|
||||
client = ScriptedChatClient(script=list(replies), sink=sink, role="proposer")
|
||||
result = asyncio.run(
|
||||
generate_via_llm(
|
||||
client,
|
||||
_project(),
|
||||
_CONTEXT,
|
||||
TokenMeter(Budget(max_tokens=100_000, max_rounds=8)),
|
||||
parse_failures=failures,
|
||||
)
|
||||
)
|
||||
return result, sink, failures
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------- C1
|
||||
|
||||
|
||||
def test_the_parse_reason_reaches_the_next_attempts_prompt() -> None:
|
||||
"""(a) The retry is no longer blind — asserted on what the CLIENT received."""
|
||||
_result, sink, _failures = _generate([_UNPARSEABLE, _VALID])
|
||||
assert len(sink) >= 2, sink
|
||||
assert "could not be PARSED" in sink[1]
|
||||
assert "claimed_saving_nok" in sink[1]
|
||||
|
||||
|
||||
def test_the_first_prompt_is_byte_identical_to_the_pre_p20_one() -> None:
|
||||
"""(b) Without this, (a) could be satisfied by always appending the block."""
|
||||
_result, sink, _failures = _generate([_VALID])
|
||||
assert "could not be PARSED" not in sink[0]
|
||||
assert sink[0] == _build_messages(_project(), _CONTEXT)[0].text
|
||||
|
||||
|
||||
def test_a_reason_does_not_survive_a_reply_that_parsed() -> None:
|
||||
"""(c) Per-RETRY, like ``prior_rejection`` is per-attempt: no stale instruction."""
|
||||
assert "could not be PARSED" not in _build_messages(_project(), "c")[0].text
|
||||
carried = _build_messages(_project(), "c", parse_error="ValueError: x")[0].text
|
||||
assert "ValueError: x" in carried
|
||||
|
||||
|
||||
def test_the_verbatim_evidence_is_still_captured() -> None:
|
||||
"""(d) Fase 1b funn 1 stands: the paid reply's own text survives the retry."""
|
||||
_result, _sink, failures = _generate([_UNPARSEABLE, _VALID])
|
||||
assert len(failures) == 1
|
||||
assert failures[0].text == _UNPARSEABLE
|
||||
assert failures[0].error.startswith("ValidationError")
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------- C2
|
||||
|
||||
|
||||
def test_the_announcement_names_the_routed_bases() -> None:
|
||||
"""(e) Two bases, two declared ids — not "the portfolio"."""
|
||||
assert announced_subject(None, (str(_TUNNEL),)) == "tunnel-hauglia"
|
||||
|
||||
|
||||
def test_an_unresolvable_base_falls_back_to_its_directory_name(tmp_path: Path) -> None:
|
||||
"""(f) Naming never changes which error an operator sees (the dimension_label precedent)."""
|
||||
missing = tmp_path / "not-a-base"
|
||||
assert announced_subject(None, (str(missing),)) == "not-a-base"
|
||||
|
||||
|
||||
def test_the_two_older_subjects_are_unchanged() -> None:
|
||||
"""(g) A named project wins; no bases at all is still the portfolio."""
|
||||
assert announced_subject("proj-1", ()) == "proj-1"
|
||||
assert announced_subject("proj-1", ("ignored",)) == "proj-1"
|
||||
assert announced_subject(None, ()) == "the portfolio"
|
||||
275
tests/test_requirement_number_gate_loadbearing.py
Normal file
275
tests/test_requirement_number_gate_loadbearing.py
Normal file
|
|
@ -0,0 +1,275 @@
|
|||
"""P20 DEL B — a clause number is not a price, and the base's own vocabulary is what says so.
|
||||
|
||||
**What was measured.** Three paid rounds and one multi-base pass carried FOUR ``validated``
|
||||
proposals whose cost code was a chapter number of a standard. Two survive in the recorded outboxes
|
||||
and are this arm's known positives:
|
||||
|
||||
* ``10.4`` — tunnel-hauglia round 3, base ``vegnormal-n500-2024``, ``validated``;
|
||||
* ``1.10.4`` — lindaas P17b, base ``vegnormal-r761-2025``, ``validated``.
|
||||
|
||||
Both are GROUNDED in P7's sense (they occur verbatim in the input) and neither is INERT in
|
||||
P18/B1's sense (``10.4`` in 12 of 274 documents, ``1.10.4`` in 1 of 2 756). Stage 0 never ran:
|
||||
no vegnormal base ships a cost baseline. Nothing in the gate could say what they are.
|
||||
|
||||
**THE ORDER'S OWN RULE WAS FELLED BY MEASUREMENT BEFORE ANYTHING WAS BUILT ON IT.** B1 reads: a
|
||||
code is a requirement when it has form 2 or 3 AND "står som ``req_number``/``prosessnr`` i
|
||||
toppnivå-frontmatter i minst ett av grunnlagets dokumenter" — refuse that. Measured 15.09:
|
||||
|
||||
* n500 declares ``seksjon: 10.4.1`` … ``10.4.4`` and ``req_number: Krav 10.4.3—2``. The bare
|
||||
``10.4`` is declared NOWHERE — it is a section PREFIX;
|
||||
* r761 declares 2 727 ``prosessnr`` and 2 753 ``seksjon``. ``1.10.4`` is NONE of them: it occurs
|
||||
once, as prose, in "Krav til materialer skal være iht. vegnormal N200 Vegbygging kap. 1.10.4".
|
||||
|
||||
The ordered rule therefore fires on NEITHER of its own known positives. The COMPLEMENT fires on
|
||||
BOTH, and it closes a hole ``_ground_against_input`` already admits in writing — "it fails OPEN …
|
||||
on a coincidental match". For one shape, a clause number, the base hands us the vocabulary needed
|
||||
to tell a real reference from a coincidence, and that is the rule built here.
|
||||
|
||||
The complement is also what SPARES the one context set built on real process codes: all five of
|
||||
``contexts/kontrakt-sorasen-2027``'s codes are declared ``prosessnr`` and pass. Under the ordered
|
||||
rule every one of them would have been refused on an unanchored r761 run, and the set's positive
|
||||
arms would have become unmeasurable — the R761 risk the order names, arriving through the door it
|
||||
was pointed away from.
|
||||
|
||||
Measured over EVERY code of round 3 and P17b (24 codes, 10 runs): exactly two are
|
||||
requirement-shaped, they are the two known positives, and the replay flips exactly those two.
|
||||
|
||||
What each arm pins:
|
||||
|
||||
(a) known positive — the ``10.4`` proposal, replayed against the base it actually ran on, is
|
||||
``rejected``, and the reason names the denominator;
|
||||
(b) known positive — ditto ``1.10.4`` on r761;
|
||||
(c) known negative — a code the base DOES declare (sorasen's real ``12.1``) still validates;
|
||||
(d) known negative — the gate is OFF when the run is anchored, even for a clause-shaped code;
|
||||
(e) the generality guard — an input that declares no reference numbers at all cannot trip the rule,
|
||||
which is what leaves every pre-P20 fixture untouched rather than exempted;
|
||||
(f) K2's identifier forms are untouched: ``SHA-01`` is not requirement-shaped;
|
||||
(g) ``classify_codes``' third value, and its denominator-free reading (``grounding=None``) that
|
||||
``stress.py`` re-derives with;
|
||||
(h) the vocabulary travels WITH the text through ``_grounding_text``, so the gate the generation
|
||||
loop runs sees what the run composed;
|
||||
(i) a run composes the vocabulary from the base it opened — asserted end-to-end through
|
||||
``run_project``, not on the composer.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser import okf
|
||||
from portfolio_optimiser.generate import _grounding_text
|
||||
from portfolio_optimiser.ir import CostBaseline, SavingsProposal
|
||||
from portfolio_optimiser.reference_domain import Project
|
||||
from portfolio_optimiser.validator import (
|
||||
Grounding,
|
||||
Rejection,
|
||||
ValidatedProposal,
|
||||
classify_codes,
|
||||
has_requirement_form,
|
||||
validate_proposal,
|
||||
)
|
||||
|
||||
_DEFAULT_BUNDLE_ROOT = Path.home() / "repos" / "vegnormal-okf" / "build" / "ferdig"
|
||||
_ROUND3 = Path("scratchpad/p19-stress/tunnel-hauglia-2027")
|
||||
_P17B = Path("scratchpad/p17b-multibase/lindaas")
|
||||
|
||||
|
||||
def _base(name: str) -> Path:
|
||||
root = Path(os.environ.get("PORTFOLIO_VEGNORMAL_ROOT", str(_DEFAULT_BUNDLE_ROOT)))
|
||||
if not (root / name).is_dir():
|
||||
pytest.skip(f"knowledge base {name!r} is not mounted under {root}")
|
||||
return root / name
|
||||
|
||||
|
||||
def _grounding_over(name: str) -> Grounding:
|
||||
"""The delivered base exactly as ``run_project`` composes it — documents AND vocabulary."""
|
||||
bundle = okf.navigate_bundle(str(_base(name)))
|
||||
return Grounding(
|
||||
documents=tuple(
|
||||
"\n".join([f.name, *f.frontmatter.values(), f.body]) for f in bundle.context_files
|
||||
),
|
||||
declared_references=tuple(
|
||||
ref for f in bundle.context_files for ref in okf.declared_reference_numbers(f)
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _recorded(path: Path) -> SavingsProposal:
|
||||
if not path.is_file():
|
||||
pytest.skip(f"the recorded artefact {path} is not present in this checkout")
|
||||
return SavingsProposal.model_validate(json.loads(path.read_text(encoding="utf-8"))["proposal"])
|
||||
|
||||
|
||||
def _proposal(code: str, *, saving: float = 1000.0) -> SavingsProposal:
|
||||
return SavingsProposal(
|
||||
project_id="p",
|
||||
measure="m",
|
||||
affected_items=[{"code": code, "quantity": 10.0, "unit_cost": 1000.0}],
|
||||
claimed_saving_nok=saving,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------- known positives
|
||||
|
||||
|
||||
def test_the_section_number_that_reached_validated_on_n500_is_refused() -> None:
|
||||
"""(a) tunnel-04's ``10.4``, replayed against the base that run actually opened."""
|
||||
proposal = _recorded(_ROUND3 / "tunnel-hauglia-2027-04-a4-enhetspris-ventilator-proposal.json")
|
||||
assert [i.code for i in proposal.affected_items] == ["10.4"]
|
||||
outcome = validate_proposal(proposal, baseline=None, grounding=_grounding_over("n500-2024"))
|
||||
assert isinstance(outcome, Rejection)
|
||||
assert "'10.4'" in outcome.reason
|
||||
assert "not one of the 365 this knowledge base declares" in outcome.reason
|
||||
|
||||
|
||||
def test_the_process_number_that_reached_validated_on_r761_is_refused() -> None:
|
||||
"""(b) lindaas a4's ``1.10.4`` — a chapter of ANOTHER standard, quoted in one r761 document."""
|
||||
proposal = _recorded(_P17B / "lindaas-01-vegnormal-r761-2025-a4-indeksregulering-proposal.json")
|
||||
assert [i.code for i in proposal.affected_items] == ["1.10.4"]
|
||||
outcome = validate_proposal(proposal, baseline=None, grounding=_grounding_over("r761-2025"))
|
||||
assert isinstance(outcome, Rejection)
|
||||
assert "not one of the 2765 this knowledge base declares" in outcome.reason
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------- known negatives
|
||||
|
||||
|
||||
def test_a_process_code_the_base_declares_still_validates() -> None:
|
||||
"""(c) The arm that keeps this a rule about the corpus and not about shapes.
|
||||
|
||||
``12.1`` is ``contexts/kontrakt-sorasen-2027``'s own first code and a REAL declared
|
||||
``prosessnr`` of R761. Under the ordered rule it would have been refused; it must not be.
|
||||
"""
|
||||
grounding = _grounding_over("r761-2025")
|
||||
assert "12.1" in grounding.reference_vocabulary
|
||||
outcome = validate_proposal(_proposal("12.1"), baseline=None, grounding=grounding)
|
||||
assert isinstance(outcome, ValidatedProposal), getattr(outcome, "reason", "")
|
||||
|
||||
|
||||
def test_every_sorasen_code_is_in_the_bases_vocabulary() -> None:
|
||||
"""(c) The whole context set, not one sample: five real codes, five declared numbers."""
|
||||
codes = [
|
||||
code
|
||||
for approach in json.loads(
|
||||
Path("contexts/kontrakt-sorasen-2027/mandate.json").read_text(encoding="utf-8")
|
||||
)["approaches"]
|
||||
for code in approach.get("affected_codes", [])
|
||||
]
|
||||
vocabulary = _grounding_over("r761-2025").reference_vocabulary
|
||||
shaped = [c for c in codes if has_requirement_form(c)]
|
||||
assert len(shaped) == 5, shaped
|
||||
assert [c for c in shaped if c not in vocabulary] == []
|
||||
|
||||
|
||||
def test_the_fasit_references_are_classified_requirement() -> None:
|
||||
"""(g) known negative (c) of the order: a fasit reference IS a requirement, and says so."""
|
||||
fasit = json.loads(
|
||||
Path("contexts/kontrakt-sorasen-2027/fasit.json").read_text(encoding="utf-8")
|
||||
)
|
||||
refs = sorted({c["ref"] for entry in fasit["must_cite"] for c in entry["concepts"]})
|
||||
forms = classify_codes(refs, _grounding_over("r761-2025"))
|
||||
assert set(forms.values()) == {"requirement"}, forms
|
||||
|
||||
|
||||
def test_an_anchored_run_is_untouched_by_the_rule() -> None:
|
||||
"""(d) Stage 0 has already ruled; the weaker stage must not overrule the stronger."""
|
||||
grounding = Grounding(documents=("12.9 is a clause",), declared_references=("12.1", "12.2"))
|
||||
baseline = CostBaseline(project_id="p", items={"12.9": {"quantity": 10.0, "unit_cost": 1000.0}})
|
||||
outcome = validate_proposal(_proposal("12.9"), baseline=baseline, grounding=grounding)
|
||||
assert isinstance(outcome, ValidatedProposal), getattr(outcome, "reason", "")
|
||||
# The control: the SAME code and the SAME text, unanchored, is refused.
|
||||
unanchored = validate_proposal(_proposal("12.9"), baseline=None, grounding=grounding)
|
||||
assert isinstance(unanchored, Rejection)
|
||||
|
||||
|
||||
def test_an_input_that_declares_no_reference_numbers_cannot_trip_the_rule() -> None:
|
||||
"""(e) The generality guard — and the reason every pre-P20 fixture is untouched."""
|
||||
grounding = Grounding(documents=("a document mentioning 12.9 once",))
|
||||
assert grounding.reference_vocabulary == frozenset()
|
||||
outcome = validate_proposal(_proposal("12.9"), baseline=None, grounding=grounding)
|
||||
assert isinstance(outcome, ValidatedProposal), getattr(outcome, "reason", "")
|
||||
|
||||
|
||||
def test_a_cost_line_identifier_is_not_requirement_shaped() -> None:
|
||||
"""(f) K2's 50 identifiers and this repo's own code: shape, measured."""
|
||||
assert not has_requirement_form("SHA-01")
|
||||
assert not has_requirement_form("ENERGI-TOTAL-EL")
|
||||
assert not has_requirement_form("65 ASFALTDEKKER")
|
||||
assert has_requirement_form("10.4") and has_requirement_form("Krav 4.1.2—1")
|
||||
|
||||
|
||||
def test_classify_codes_without_a_grounding_is_the_pre_p20_answer() -> None:
|
||||
"""(g) ``stress.py`` re-derives for runs written before the field existed."""
|
||||
assert classify_codes(["12.1", "SHA-01", "impulsventilator"]) == {
|
||||
"12.1": "identifier",
|
||||
"SHA-01": "identifier",
|
||||
"impulsventilator": "prose",
|
||||
}
|
||||
grounding = Grounding(documents=("x",), declared_references=("12.1",))
|
||||
assert classify_codes(["12.1"], grounding) == {"12.1": "requirement"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------- the wiring
|
||||
|
||||
|
||||
def test_the_vocabulary_travels_with_the_text_into_the_generation_gate() -> None:
|
||||
"""(h) ``_grounding_text`` composes the run's three sources; the vocabulary must survive it."""
|
||||
delivered = Grounding(documents=("d",), declared_references=("12.1",))
|
||||
project = Project(
|
||||
id="p", name="n", description="d", currency="NOK", cost_items=(), docs_dir="."
|
||||
)
|
||||
composed = _grounding_text(project, None, delivered)
|
||||
assert composed.reference_vocabulary == frozenset({"12.1"})
|
||||
|
||||
|
||||
def test_a_run_composes_the_vocabulary_from_the_base_it_opened(tmp_path: Path) -> None:
|
||||
"""(i) End-to-end through ``run_project``: the stamp says ``requirement`` for a declared code.
|
||||
|
||||
Asserted on the ARTEFACT a run leaves, never on the composer — a vocabulary wired nowhere would
|
||||
satisfy every arm above and none of this one.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
from agent_framework import BaseChatClient
|
||||
|
||||
from portfolio_optimiser.run import run_project
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
|
||||
base = tmp_path / "mini"
|
||||
base.mkdir()
|
||||
(base / "index.md").write_text(
|
||||
"---\nbundle_id: mini\n---\n\n- [Krav](krav.md) — one clause.\n", encoding="utf-8"
|
||||
)
|
||||
(base / "krav.md").write_text(
|
||||
"---\ntype: Krav\ntitle: Krav 4.1.2-1\nprosessnr: '12.1'\n---\n\nEn kostlinje 12.1.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(base / "validator-input.json").write_text(
|
||||
json.dumps({"project_id": "mini-p", "measure": "m", "affected_codes": ["12.1"]}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
reply = (
|
||||
'{"measure":"m","affected_items":[{"code":"12.1","quantity":10,"unit_cost":1000}],'
|
||||
'"claimed_saving_nok":1000}'
|
||||
)
|
||||
|
||||
def factory(role: str) -> BaseChatClient:
|
||||
return ScriptedChatClient(
|
||||
"Reasoning holds.\nVERDICT: APPROVE" if role == "checker" else reply, role=role
|
||||
)
|
||||
|
||||
result = asyncio.run(
|
||||
run_project(
|
||||
"mini-p",
|
||||
"local",
|
||||
docs_dir=str(base),
|
||||
bundle_dir=str(base),
|
||||
client_factory=factory,
|
||||
)
|
||||
)
|
||||
assert result.provenance.code_forms == {"12.1": "requirement"}
|
||||
assert result.provenance.validator_decision == "validated"
|
||||
234
tests/test_right_requirement_loadbearing.py
Normal file
234
tests/test_right_requirement_loadbearing.py
Normal file
|
|
@ -0,0 +1,234 @@
|
|||
"""P20 DEL A — the requirement that is RIGHT, and the commission's criteria reaching the reader.
|
||||
|
||||
**The measured silence, three rounds and one multi-base pass deep.** P19 DEL A made a direction
|
||||
NAME the requirement that binds it and made the run refuse a declaration naming a path it never
|
||||
opened. The declarations then happened — and MEASURED (P19 round 3: 9 declarations over 5 paid
|
||||
runs; P17b: 4 over one two-base pass) **not one of them named a fasit concept**. The rung works;
|
||||
what nothing asked for was that the requirement be the RIGHT one. Two halves were missing:
|
||||
|
||||
* the tool answered ``{"declared": true, ...}`` to every accepted declaration, echoing back the
|
||||
caller's own three arguments. A model that had declared a requirement about something else was
|
||||
told, in the only words it got, that it had succeeded;
|
||||
* the commission's own statement of what a good answer looks like — ``Mandate.success_criteria`` —
|
||||
reached ``announce`` and NOTHING else (P19 F2). It was printed for a human and withheld from the
|
||||
only reader who could act on it.
|
||||
|
||||
What each arm pins:
|
||||
|
||||
(a) A1 — the reply carries the DOCUMENT's own ``title`` and ``req_number``, read off the base, plus
|
||||
the sentence that says what the declaration BINDS. Without the document's own words the reply
|
||||
cannot contradict a wrong declaration, which is the whole correction;
|
||||
(b) A1 — a path the base carries as no navigated concept (``index.md`` is the reachable case)
|
||||
answers with empty strings rather than raising: the read trace has already accepted the
|
||||
declaration, and turning "I cannot restate your title" into a refusal would fail a declaration
|
||||
the run's own evidence proves was read;
|
||||
(c) A1 — the reply is read off ``Bundle.context_files``, so a ``type: verdict`` document can never
|
||||
be named back by title. The one layer no listing names stays unnamed even in an answer;
|
||||
(d) A2 — ``criteria_block`` is the ONE renderer, and it is EMPTY without criteria. That omission is
|
||||
what keeps every prompt of every un-commissioned run byte-identical, the demo's included;
|
||||
(e) A2 — the criteria reach the DEBATE's task message, the prompt where ``declare_requirement`` is
|
||||
available. Asserted on the text the client actually received, never on the composer;
|
||||
(f) A2 — a run with NO mandate sends the task message byte-identically to before, which is the
|
||||
half that keeps ``demo-transcript.stdout`` unchanged and is asserted here rather than left to
|
||||
the golden;
|
||||
(g) A3 — the navigator instruction and the tool description both name the ``read_dir(filter=...)``
|
||||
route with a worked example. A description that lies about the body IS the model's instruction
|
||||
(the Fase-3 class), and here the body is new.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser import explore, okf
|
||||
from portfolio_optimiser.explore import DeclaredRequirement, ToolCall, navigator_tools
|
||||
from portfolio_optimiser.mandate import Approach, Mandate, criteria_block
|
||||
|
||||
_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
|
||||
_TUNNEL = _EXAMPLES / "tunnel-hauglia"
|
||||
_BYGG = _EXAMPLES / "bygg-energi-mikro"
|
||||
_PID = "BYGG-KONTOR-NORD"
|
||||
|
||||
|
||||
def _wired(bundle_dir: Path) -> tuple[dict[str, Any], list[ToolCall], list[DeclaredRequirement]]:
|
||||
opened: list[ToolCall] = []
|
||||
declared: list[DeclaredRequirement] = []
|
||||
tools = navigator_tools((str(bundle_dir),), opened=opened, requirements=declared)
|
||||
return {t.name: t for t in tools}, opened, declared
|
||||
|
||||
|
||||
def _declare(tools: dict[str, Any], opened: list[ToolCall], base: str, path: str) -> dict[str, Any]:
|
||||
"""Read it the way a run does, then declare it — the (b) path of P19 DEL A."""
|
||||
opened.append(ToolCall(name="read_file", bundle_id=base, path=path))
|
||||
return tools["declare_requirement"].func(bundle_id=base, path=path, ref="Krav 4.1.2-1")
|
||||
|
||||
|
||||
def _base_with_a_requirement(root: Path) -> Path:
|
||||
"""A base declaring its own ``req_number`` — the form the delivered N corpora carry.
|
||||
|
||||
Crafted rather than taken from ``shared/examples``: MEASURED, no example base declares a
|
||||
``req_number`` at all, so an arm built on one of them could not tell a reply that reads the
|
||||
document from one that returns an empty string.
|
||||
"""
|
||||
base = root / "n-mini"
|
||||
base.mkdir()
|
||||
(base / "index.md").write_text(
|
||||
"---\nbundle_id: n-mini\n---\n\n- [Krav](krav.md) — one requirement.\n", encoding="utf-8"
|
||||
)
|
||||
(base / "krav.md").write_text(
|
||||
"---\ntype: Krav\ntitle: Krav 4.1.2-1 Rundkjoring\nreq_number: Krav 4.1.2-1\n"
|
||||
"seksjon: '4.1.2'\n---\n\nEn rundkjoring skal ha …\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return base
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# (a)-(c) the reply the model can be contradicted by
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_reply_carries_the_documents_own_title_and_number(tmp_path: Path) -> None:
|
||||
"""(a) The base's words, not the caller's — the only thing that can say "wrong one"."""
|
||||
base = _base_with_a_requirement(tmp_path)
|
||||
tools, opened, declared = _wired(base)
|
||||
answer = _declare(tools, opened, "n-mini", "krav.md")
|
||||
assert answer["title"] == "Krav 4.1.2-1 Rundkjoring"
|
||||
assert answer["req_number"] == "Krav 4.1.2-1"
|
||||
assert "is the requirement the proposal rests on" in answer["binds"]
|
||||
assert declared == [DeclaredRequirement(bundle_id="n-mini", path="krav.md", ref="Krav 4.1.2-1")]
|
||||
|
||||
|
||||
def test_a_path_that_is_no_concept_answers_with_empty_strings(tmp_path: Path) -> None:
|
||||
"""(b) Accepted-but-unrestatable is not a refusal; the read trace already ruled."""
|
||||
base = _base_with_a_requirement(tmp_path)
|
||||
tools, opened, _declared = _wired(base)
|
||||
answer = _declare(tools, opened, "n-mini", "index.md")
|
||||
assert answer["declared"] is True
|
||||
assert (answer["title"], answer["req_number"]) == ("", "")
|
||||
|
||||
|
||||
def test_a_verdict_document_is_never_named_back_by_title(tmp_path: Path) -> None:
|
||||
"""(c) Read off ``context_files``, the property that drops the ``type: verdict`` layer."""
|
||||
base = _base_with_a_requirement(tmp_path)
|
||||
(base / "dom.md").write_text(
|
||||
"---\ntype: verdict\ntitle: A PRIOR EXPERT JUDGEMENT\nid: v1\n---\n\napproved.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
tools, opened, _declared = _wired(base)
|
||||
answer = _declare(tools, opened, "n-mini", "dom.md")
|
||||
assert answer["declared"] is True
|
||||
assert answer["title"] == "", "the verdict layer must not be readable through this answer"
|
||||
# The control: the SAME base answers the concept's title, so the empty string above is the
|
||||
# layer being excluded rather than the reader being broken.
|
||||
assert _declare(tools, opened, "n-mini", "krav.md")["title"] == "Krav 4.1.2-1 Rundkjoring"
|
||||
|
||||
|
||||
def test_the_tunnel_example_answers_its_own_title_and_no_number() -> None:
|
||||
"""(a) control on a REAL example base: title present, reference number honestly absent."""
|
||||
tools, opened, _declared = _wired(_TUNNEL)
|
||||
path = okf.navigate_bundle(str(_TUNNEL)).context_files[0].name
|
||||
answer = _declare(tools, opened, "tunnel-hauglia", path)
|
||||
assert answer["title"].startswith("Kilder: tunnelbelysning")
|
||||
assert answer["req_number"] == "", "this base declares none, and the reply must not invent one"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# (d)-(f) the criteria reaching the prompt
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_criteria_renderer_is_empty_without_criteria() -> None:
|
||||
"""(d) Omission, never an empty heading — the ``announce`` rule, and the golden's guarantee."""
|
||||
assert criteria_block("") == ""
|
||||
rendered = criteria_block("a saving the price schedule can carry")
|
||||
assert "a saving the price schedule can carry" in rendered
|
||||
assert rendered.startswith("\n")
|
||||
|
||||
|
||||
def test_the_commissions_criteria_reach_the_debate_task(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""(e) Asserted on what the CLIENT received, never on the composer."""
|
||||
sent = _run_and_capture_task(
|
||||
monkeypatch,
|
||||
mandate=Mandate(
|
||||
objective="cut cost",
|
||||
approaches=(Approach(id="a1", label="L", description="D"),),
|
||||
success_criteria="SENTINEL-CRITERION-ONLY-THE-COMMISSION-CARRIES",
|
||||
),
|
||||
)
|
||||
assert "SENTINEL-CRITERION-ONLY-THE-COMMISSION-CARRIES" in sent[0]
|
||||
assert "What the commissioner counts as success" in sent[0]
|
||||
|
||||
|
||||
def test_a_run_without_a_mandate_sends_the_task_unchanged(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""(f) The byte-identity half. Without it (e) could be satisfied by always appending."""
|
||||
sent = _run_and_capture_task(monkeypatch, mandate=None)
|
||||
assert sent[0].startswith(f"Find a cost-saving measure for {_PID}.\nContext:\n")
|
||||
|
||||
|
||||
def _run_and_capture_task(monkeypatch: pytest.MonkeyPatch, *, mandate: Mandate | None) -> list[str]:
|
||||
"""Drive a real ``run_project`` and capture every prompt the debate's clients received.
|
||||
|
||||
The same recording-factory seam ``test_debate_navigation_cost_loadbearing`` measures S2c with:
|
||||
the assertion is on what the CLIENT was handed, never on ``criteria_block``'s return value, so
|
||||
a renderer wired nowhere cannot satisfy it.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
from agent_framework import BaseChatClient
|
||||
|
||||
from portfolio_optimiser.run import run_project
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
|
||||
sink: list[str] = []
|
||||
valid = (
|
||||
'{"measure":"LED-retrofit","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],'
|
||||
'"claimed_saving_nok":30000}'
|
||||
)
|
||||
|
||||
def factory(role: str) -> BaseChatClient:
|
||||
client = ScriptedChatClient(
|
||||
"Reasoning holds.\nVERDICT: APPROVE" if role == "checker" else valid, role=role
|
||||
)
|
||||
original = client._inner_get_response # type: ignore[attr-defined]
|
||||
|
||||
def recording(*, messages, options, stream=False, **kwargs): # type: ignore[no-untyped-def]
|
||||
sink.append("\n".join(getattr(m, "text", "") or "" for m in messages))
|
||||
return original(messages=messages, options=options, stream=stream, **kwargs)
|
||||
|
||||
client._inner_get_response = recording # type: ignore[attr-defined,method-assign]
|
||||
return client
|
||||
|
||||
asyncio.run(
|
||||
run_project(
|
||||
_PID,
|
||||
"local",
|
||||
docs_dir=str(_BYGG),
|
||||
bundle_dir=str(_BYGG),
|
||||
verdict_input={"decision": "approved", "rationale": "expert reviewed (test)"},
|
||||
client_factory=factory,
|
||||
mandate=mandate,
|
||||
)
|
||||
)
|
||||
assert sink, "the debate never reached the client"
|
||||
return sink
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# (g) the instruction that describes the new body
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_both_descriptions_name_the_filter_route() -> None:
|
||||
"""(g) A description that lies about the body IS the instruction (the Fase-3 class)."""
|
||||
instruction = explore._INSTRUCTIONS[explore.HYPOTHESISER_ROLE]
|
||||
assert "read_dir a 'filter' word" in instruction
|
||||
assert "Krav 4.1.2-1" in instruction, "the worked example is what makes the route concrete"
|
||||
tools = {t.name: t for t in navigator_tools((str(_TUNNEL),), opened=[], requirements=[])}
|
||||
description = tools["declare_requirement"].description or ""
|
||||
assert "filter='rundkjoring'" in description
|
||||
assert "own title and number" in description
|
||||
Loading…
Add table
Add a link
Reference in a new issue