P6 (økt 108) ended in ValidatedProposal (verdict 5fd6272e3725fe68) on two cost codes -- M-04-01 / M-04-03 -- that appear in NO prompt of that run. Measured here first, verbatim: validate_proposal(p, baseline=None) validates it; the same proposal against any non-empty CostBaseline is rejected naming both codes. So the hole was never "fabrication goes uncaught" -- _reconcile_against_baseline exists and is right -- but that the falsifier is reached only through `if baseline is not None`. The input always exists; the baseline does not. New stage 0b (_ground_against_input), OUTSIDE the baseline branch, after stage 0 so an anchored run's message is byte-identical to before. ONE Rejection, the validator's own type, naming EVERY ungrounded identifier "; "-joined in the proposal's own order (økt 94's completeness reason). The rule has NO pattern -- `code in grounding`, exact substring -- and that is a measurement: over the delivered corpora (K2 1108 files / 2 005 561 chars, the three N payloads 8 excerpts each) the identifier forms are heterogeneous, and a pattern chosen to cover them would be a rule about shapes. Bare numerals are the one inert class (46 394 occurrences / 2 117 distinct in K2); the rule fails OPEN there, never closed. Evidence is three non-model-authored sources: what run_project DELIVERED (the rendered cut/pointer/chunks plus the base's context_files -- never files, which would make the type: verdict layer evidence), the project's own cost lines, and the baseline's codes when anchored. The rendered PROMPT is deliberately NOT evidence, on two measurements: gen_context IS the debate output on the S2c path, and from attempt 2 the prompt carries the previous Rejection.reason verbatim -- which for this stage QUOTES the identifier it just refused. Grounding in the prompt would let the gate's own refusal disarm it on its second round. Prose scanning was chosen against WITH THE NUMBERS: a typed gate catches 2/2 (P6) and 2/2 (S7c) -- 100% of what reached a verdict. What stays uncaught, said plainly: an ungrounded identifier that lives only in agent/debate prose and never becomes an affected_item code (2 of 4 P6, 2 of 4 S7c, 1 of 2 P4). Iron Law: 9 red / 2 green before the rule existed. Ten mutations all red against the whole suite, green control 1558 passed / 5 skipped (from 1543/5, superset, 0 removed), golden demo-transcript.stdout BYTE-UNCHANGED (shasum -a 1 of the CONTENT = ea8c534773acdbe41ae68f2c55724d69aaf8be4f). Three existing fixtures changed, no gate weakened -- most of all test_pre_amendment_bundle_runs_unchanged, which sent the SAME FABRICATED code and asserted it validated: the økt-108 hole written down as an expectation. No paid run. Order 20260909T113641Z-38938691-from-.claude. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
213 lines
9.6 KiB
Python
213 lines
9.6 KiB
Python
"""Step 2 load-bearing seam (SC2 + brief §4.1, målbilde §2/§6): a run's ``dimension`` does TWO
|
|
things, each with a named detach point:
|
|
|
|
- **Candidate constraint (§4.1b):** a validator-VALID candidate (P90-valid, empty assumptions)
|
|
whose ``measure_type`` falls OUTSIDE the run's dimension is rejected WHEN a dimension is set.
|
|
The proposal validates on the numbers, so the ONLY possible rejecter is the ``admits`` scope
|
|
gate (closes the green-but-dead trap). RED if ``admits`` is removed ⇒ the foreign candidate
|
|
slips through. Control: an in-dimension candidate passes.
|
|
- **Context scope (§4.1a), as it stands after S2c:** the promise is unchanged — a dimension-scoped
|
|
run lets the agents read ONLY dimension-matched bundle knowledge — but the debate no longer
|
|
RECEIVES a rendered context, it NAVIGATES the base, so the filter had to move from
|
|
``bundle_context`` to the tools the debate is handed. These two arms gate the FIRST rung: the
|
|
listing the debate can see never names a foreign-dimension document. RED if the ``dimension=``
|
|
arg is dropped anywhere along ``run_project`` → ``navigator_tools`` → ``directory_listing``.
|
|
Control: ``dimension=None`` ⇒ the same document IS listed. The SECOND rung (``read_file``
|
|
refusing a foreign document by path) is gated in
|
|
``tests/test_debate_navigation_cost_loadbearing.py`` — a listing filter with an ungated reader
|
|
behind it is a filter in name only, so the two halves get their own arms and their own
|
|
mutations.
|
|
|
|
Patterns: ``test_checker_gate_loadbearing.py:59/87`` (gate + causality control),
|
|
``conftest.py:184`` (recording client), ``test_step8_promotion_loadbearing.py:51`` (bundle copy).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import shutil
|
|
from collections.abc import Callable
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
from agent_framework import BaseChatClient
|
|
from conftest import SyntheticUsageChatClient
|
|
|
|
from portfolio_optimiser import okf
|
|
from portfolio_optimiser.dimension import Dimension
|
|
from portfolio_optimiser.run import run_project
|
|
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
|
|
|
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
|
|
|
_ENERGY_DIM = Dimension(
|
|
id="energi", label="Energi", allowed_measure_types=frozenset({"energy_efficiency"})
|
|
)
|
|
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
|
|
|
|
# A marker that appears ONLY in the asfalt-marked concept file's body, so it can reach the prompt
|
|
# solely through un-filtered context — its presence/absence is the §4.1a leak probe.
|
|
_ASFALT_SENTINEL = "ASFALT-LEAK-SENTINEL-x7y8z9"
|
|
|
|
|
|
def _valid_reply(measure: str, code: str) -> str:
|
|
"""A validator-VALID proposal: affected total 300000, degenerate P90 = 0.30 x 300000 = 90000
|
|
>= claimed 30000, empty assumptions -> validates on the numbers regardless of ``measure``/``code``
|
|
(so a rejection can only come from the dimension scope gate)."""
|
|
return (
|
|
'{"measure":"' + measure + '","affected_items":'
|
|
'[{"code":"' + code + '","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
|
|
)
|
|
|
|
|
|
def _role_factory(proposer_reply: str, checker_reply: str) -> Callable[[str], BaseChatClient]:
|
|
def factory(role: str) -> BaseChatClient:
|
|
return SyntheticUsageChatClient(
|
|
default_reply=checker_reply if role == "checker" else proposer_reply
|
|
)
|
|
|
|
return factory
|
|
|
|
|
|
# --- Candidate constraint (§4.1b) ----------------------------------------------------------------
|
|
|
|
|
|
async def test_foreign_dimension_candidate_rejected_when_dimension_set() -> None:
|
|
"""LOAD-BEARING: a foreign-dimension candidate that validates on the numbers is rejected by the
|
|
scope gate. RED if ``admits`` is detached (the foreign candidate slips through as validated).
|
|
|
|
The cost code is one the delivered base NAMES (P7). It used to be ``SENTINEL-FOREIGN``, which
|
|
the base does not carry — so once identifier grounding existed the candidate fell for a reason
|
|
that has nothing to do with the dimension, and this arm would have ridden on that instead of on
|
|
``admits``. Only the MEASURE is foreign now, which is what §4.1b is about."""
|
|
factory = _role_factory(
|
|
_valid_reply("paving_renegotiation", "ENERGI-TOTAL-EL"), "VERDICT: APPROVE"
|
|
)
|
|
|
|
result = await run_project(
|
|
"BYGG-KONTOR-NORD",
|
|
"local",
|
|
docs_dir=str(BUNDLE_DIR),
|
|
bundle_dir=str(BUNDLE_DIR),
|
|
verdict_input=_VERDICT_INPUT,
|
|
dimension=_ENERGY_DIM,
|
|
client_factory=factory,
|
|
)
|
|
|
|
assert isinstance(result.outcome, Rejection), (
|
|
"a foreign-dimension candidate slipped through — the admits scope gate is not gating"
|
|
)
|
|
assert "outside dimension" in result.outcome.reason
|
|
# Provenance honesty: the VALIDATOR passed (the numbers are feasible); only the scope gate
|
|
# rejected. validator_decision reflects the numbers ONLY — never the scope gate.
|
|
assert result.provenance.validator_decision == "validated"
|
|
|
|
|
|
async def test_in_dimension_candidate_passes() -> None:
|
|
"""CAUSALITY CONTROL: the SAME shape with an in-dimension ``measure_type`` validates normally —
|
|
proving the rejection above is caused by the dimension scope, not the fixture."""
|
|
factory = _role_factory(
|
|
_valid_reply("energy_efficiency", "ENERGI-TOTAL-EL"), "VERDICT: APPROVE"
|
|
)
|
|
|
|
result = await run_project(
|
|
"BYGG-KONTOR-NORD",
|
|
"local",
|
|
docs_dir=str(BUNDLE_DIR),
|
|
bundle_dir=str(BUNDLE_DIR),
|
|
verdict_input=_VERDICT_INPUT,
|
|
dimension=_ENERGY_DIM,
|
|
client_factory=factory,
|
|
)
|
|
|
|
assert isinstance(result.outcome, ValidatedProposal)
|
|
|
|
|
|
# --- Context scope (§4.1a) -----------------------------------------------------------------------
|
|
|
|
|
|
def _bundle_with_asfalt_file(tmp_path: Path) -> str:
|
|
"""A throwaway copy of the shared bundle with one extra asfalt-marked concept file carrying the
|
|
sentinel in its body, linked from the index so ``navigate_bundle`` reaches it. The shared,
|
|
framework-neutral fixture is never mutated (mirrors ``_copy_bundle``)."""
|
|
dst = tmp_path / "bundle"
|
|
shutil.copytree(BUNDLE_DIR, dst)
|
|
(dst / "asfalt-note.md").write_text(
|
|
f"---\ntype: methodology\ndimension: asfalt\n---\n\n{_ASFALT_SENTINEL} — paving method note\n",
|
|
encoding="utf-8",
|
|
)
|
|
index = dst / "index.md"
|
|
index.write_text(
|
|
index.read_text(encoding="utf-8") + "\n- [asfalt](asfalt-note.md)\n", encoding="utf-8"
|
|
)
|
|
return str(dst)
|
|
|
|
|
|
async def _listing_the_debate_can_see(bundle_dir: str, dimension: Dimension | None) -> str:
|
|
"""Run the bundle path, capture the tools the debate was built with, and return what its
|
|
``read_bundle`` rung answers — the S2c successor to reading the prompt text.
|
|
|
|
The tool is invoked DIRECTLY afterwards rather than through an agent turn, because a
|
|
``ScriptedChatClient`` returns text and never emits a ``function_call``: a test that only drove
|
|
the run would leave the entire tool surface outside the gate (``test_explore_loadbearing``'s
|
|
own measured correction)."""
|
|
import portfolio_optimiser.run as run_module
|
|
|
|
captured: list[list[object]] = []
|
|
original = run_module.fresh_workflow
|
|
|
|
def spy(*args: object, **kwargs: object) -> object:
|
|
captured.append(list(kwargs.get("tools") or [])) # type: ignore[arg-type]
|
|
return original(*args, **kwargs) # type: ignore[arg-type]
|
|
|
|
def factory(role: str) -> BaseChatClient:
|
|
return SyntheticUsageChatClient(
|
|
default_reply=_valid_reply("energy_efficiency", "ENERGI-TOTAL-EL")
|
|
)
|
|
|
|
monkeypatch = pytest.MonkeyPatch()
|
|
try:
|
|
monkeypatch.setattr(run_module, "fresh_workflow", spy)
|
|
await run_project(
|
|
"BYGG-KONTOR-NORD",
|
|
"local",
|
|
docs_dir=bundle_dir,
|
|
bundle_dir=bundle_dir,
|
|
verdict_input=_VERDICT_INPUT,
|
|
dimension=dimension,
|
|
client_factory=factory,
|
|
)
|
|
finally:
|
|
monkeypatch.undo()
|
|
|
|
assert captured, "the debate was never built"
|
|
read_bundle = next(t for t in captured[0] if getattr(t, "name", "") == "read_bundle")
|
|
bundle_id = okf.reconcile_bundle_id(bundle_dir).id
|
|
answer = await read_bundle.invoke(arguments={"bundle_id": bundle_id})
|
|
return "".join(getattr(c, "text", "") or "" for c in answer)
|
|
|
|
|
|
async def test_dimension_scopes_the_agent_context(tmp_path) -> None:
|
|
"""LOAD-BEARING (§4.1a, first rung): under a dimension the listing the debate can see does NOT
|
|
name the foreign-dimension document. RED if ``dimension=`` is dropped anywhere between
|
|
``run_project`` and ``directory_listing``.
|
|
|
|
Asserts on the FILE NAME rather than the body sentinel, because a listing carries names and
|
|
sizes, never bodies — an assert on the sentinel would be green against every implementation
|
|
and would prove nothing (the vacuous form this repo keeps measuring)."""
|
|
bundle_dir = _bundle_with_asfalt_file(tmp_path)
|
|
|
|
scoped = await _listing_the_debate_can_see(bundle_dir, _ENERGY_DIM)
|
|
assert "asfalt-note.md" not in scoped, (
|
|
"another dimension's document is listed to the agents — the tools are not dimension-scoped"
|
|
)
|
|
|
|
|
|
async def test_no_dimension_leaves_context_unscoped(tmp_path) -> None:
|
|
"""CAUSALITY CONTROL: with ``dimension=None`` the asfalt document IS listed — proving its
|
|
absence above is caused by the dimension scope, not by the file being unreachable."""
|
|
bundle_dir = _bundle_with_asfalt_file(tmp_path)
|
|
|
|
unscoped = await _listing_the_debate_can_see(bundle_dir, None)
|
|
assert "asfalt-note.md" in unscoped, (
|
|
"the asfalt file is unreachable even without a filter — the control does not prove causality"
|
|
)
|