Kjell Tore Guttormsen
938a1ca30e
Stress round 6 validated three falsification arms, and every validated approach rested only on run-level declarations nobody can attribute to one approach. declare_requirement now takes a required approach_id (a mandate id or own-proposal; an unknown id is refused naming the valid ones), and a ValidatedProposal whose approach has neither a mandate requirement nor a declaration under its own id becomes validator.Unsupported - a Rejection subclass carrying the validator's own ruling, reported as `unsupported` in coverage, the outcome artefact, the settlement and the judge, and never counted or summed. The rule is active whenever the debate held the declaration tool, the micro base included; the road and pre-pass paths are untouched. Declaration quality is not judged, so the rule can be satisfied by declaring any document the run read. The v1 gate's row 6 probes pass; its artefact half reads IKKE MÅLT because stress round 6 predates approach-addressed declarations, and IKKE MÅLT is never green - it fails the exit code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
438 lines
19 KiB
Python
438 lines
19 KiB
Python
"""P19 DEL A — a direction must NAME the requirement that binds it, and it must have READ it.
|
|
|
|
**The measured silence, two paid rounds deep.** P16 (`docs/2026-09-12-p14-kontekstsett.md` § 4.1,
|
|
`docs/2026-09-14-p18-stressrunde-2.md`) and P18 both scored **0 of 26** fasit concepts opened —
|
|
the same number twice, over four and then five paid runs, while every run still produced proposals
|
|
the deterministic gate then judged. P18 closed the navigation side of it (a listing is a window; an
|
|
invented path is refused by name) and the number did not move at all. That is what turns it from a
|
|
LADDER question into a ROLE one: nothing anywhere in the loop ever asked the model to say what
|
|
requirement of the corpus binds the direction it committed to, so opening one was never on the
|
|
critical path of producing an answer.
|
|
|
|
**A PREMISE OF THE ORDER WAS FELLED BEFORE ANYTHING WAS BUILT ON IT.** A1 places the demand in
|
|
``_INSTRUCTIONS[HYPOTHESISER_ROLE]`` alone. MEASURED: the stress command (P18 D1, repeated
|
|
verbatim in P19 E1) passes ``--mandate`` and NOT ``--explore``, the two are refused together by
|
|
name, and not one of the nine round-1/round-2 outboxes holds a ``{run_id}-exploration.json``. The
|
|
hypothesiser therefore never runs in a stress round, and a demand that only it can carry would be
|
|
structurally inert in exactly the paid runs this order commissions — A3 ("the requirement reaches
|
|
the proposal") unreachable along with it.
|
|
|
|
A2's own sentence is what resolves it: the refusal must go to the model "som en tur den kan rette
|
|
(samme mekanisme som ``quick_validate``s nekt), ikke som en ``raise``". ``quick_validate`` is a
|
|
TOOL. So the demand is a tool — ``declare_requirement`` — and it lives in ``navigator_tools``,
|
|
which since S2c is held by BOTH roles that navigate: the exploration's navigator/hypothesiser and
|
|
the debate's proposer/checker. One instruction, one refusal, one record, two doors.
|
|
|
|
What each arm pins, and what it refuses:
|
|
|
|
(a) the refusal itself — a declaration naming a path this run never opened comes back REFUSED, in
|
|
the funn-99 form (``refusal`` carries the kind ``RequirementNotRead``), and NOTHING is
|
|
recorded. Without it the tool accepts any string and the whole demand is decorative;
|
|
(b) the correction works — the same path, once ``read_file`` has actually returned it, is accepted
|
|
and recorded. A gate that can only refuse is the mirror image of one that can only pass, and
|
|
both prove nothing;
|
|
(c) an unknown base is refused by the SAME index every read rung uses, so a declaration cannot
|
|
name a corpus this run was never given;
|
|
(d) the tool EXISTS only when the caller offers both sinks — which is what keeps every other
|
|
``navigator_tools`` call site byte-identical — and passing one without the other is refused at
|
|
construction, because a sink with no read trace would accept every declaration (the repo's
|
|
vacuous-gate class, stated in advance rather than discovered);
|
|
(e) the marked hypothesis carries ``requirement`` as a REQUIRED key: omitted is a hard error of the
|
|
same class as an unreadable marked line, explicit ``null`` is legal and needs ``why_none``, and
|
|
a half-named requirement is refused (validation, never repair);
|
|
(f) a MINTED approach carries it and a SEED does not acquire one — § C.6 door 1 is a preservation
|
|
rule, and filling the field in on an expert's behalf would put their name on a claim about the
|
|
corpus they did not make;
|
|
(g) A3 — the requirement reaches the proposer's prompt VERBATIM, and a prompt whose approach has
|
|
none is byte-identical to before. That second half is what keeps ``demo-transcript.stdout``
|
|
unchanged, and it is asserted here rather than left to the golden;
|
|
(h) A4 — the judge counts a hit against THIS APPROACH'S fasit concepts, never against the base. A
|
|
judge matching the whole base would mark every declaration a hit on a 2 756-document corpus,
|
|
which is the P16 vacuity ``citation_scope`` already exists to refuse;
|
|
(i) the declaration reaches ``{run_id}-debate.json``, so a paid run's evidence survives the process
|
|
that produced it.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
from portfolio_optimiser import explore, stress
|
|
from portfolio_optimiser.explore import (
|
|
DeclaredRequirement,
|
|
HypothesisParseError,
|
|
ToolCall,
|
|
navigator_tools,
|
|
)
|
|
from portfolio_optimiser.generate import _build_messages
|
|
from portfolio_optimiser.mandate import Approach, BindingRequirement
|
|
from portfolio_optimiser.reference_domain import Project
|
|
|
|
_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
|
|
_TUNNEL = _EXAMPLES / "tunnel-hauglia"
|
|
|
|
|
|
def _wired(
|
|
bundle_dir: Path = _TUNNEL,
|
|
) -> tuple[dict[str, Any], list[ToolCall], list[DeclaredRequirement]]:
|
|
"""The tools as a RUN holds them: the declaration rung reading the recorder's own trace."""
|
|
opened: list[ToolCall] = []
|
|
declared: list[DeclaredRequirement] = []
|
|
tools = navigator_tools((str(bundle_dir),), opened=opened, requirements=declared)
|
|
return {t.name: t for t in tools}, opened, declared
|
|
|
|
|
|
def _a_concept(bundle_dir: Path = _TUNNEL) -> str:
|
|
"""One real concept path in the base, taken from the navigated listing rather than guessed."""
|
|
from portfolio_optimiser import okf
|
|
|
|
return okf.navigate_bundle(str(bundle_dir)).context_files[0].name
|
|
|
|
|
|
# ---------------------------------------------------------------------------------------------
|
|
# (a)-(c) the tool
|
|
# ---------------------------------------------------------------------------------------------
|
|
|
|
|
|
def test_a_requirement_the_run_never_opened_is_refused_and_recorded_nowhere() -> None:
|
|
"""(a) The declaration's one falsifier is the run's own read trace."""
|
|
tools, _opened, declared = _wired()
|
|
answer = tools["declare_requirement"].func(
|
|
approach_id="a1", bundle_id="tunnel-hauglia", path=_a_concept(), ref="12.1"
|
|
)
|
|
assert answer["refusal"] == "RequirementNotRead"
|
|
assert "0 document(s)" in answer["refused"]
|
|
assert declared == [], "a refused declaration must leave no record"
|
|
|
|
|
|
def test_the_correction_is_to_read_it_and_then_it_is_accepted() -> None:
|
|
"""(b) The refusal is a turn the model can correct — the half that makes (a) a gate.
|
|
|
|
P21/C1 added a SECOND precondition — the run must have opened at least three distinct
|
|
documents (capped by the base's own size) — so the correction is now "read it, and read enough
|
|
of the base to have looked". The arm keeps its meaning: a declaration the run's own trace
|
|
supports is ACCEPTED and RECORDED.
|
|
"""
|
|
tools, opened, declared = _wired()
|
|
from portfolio_optimiser import okf
|
|
|
|
files = [f.name for f in okf.navigate_bundle(str(_TUNNEL)).context_files]
|
|
path = files[0]
|
|
body = tools["read_file"].func(bundle_id="tunnel-hauglia", path=path)
|
|
assert not body.startswith("REFUSED"), body[:120]
|
|
# The recorder is middleware in a real run; here the trace is appended directly, which is the
|
|
# SAME list the tool reads.
|
|
for other in files[1:3]:
|
|
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=other))
|
|
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
|
|
answer = tools["declare_requirement"].func(
|
|
approach_id="a1", bundle_id="tunnel-hauglia", path=path, ref="Krav 12.1"
|
|
)
|
|
# P20/A1 widened the reply: the three arguments PLUS the document's own title and number and
|
|
# the sentence saying what the declaration binds. Asserted key by key rather than by equality,
|
|
# because an exact-dict assert here would fail on every future field while saying nothing about
|
|
# the one property this arm exists for — that the declaration was ACCEPTED and RECORDED.
|
|
assert answer["declared"] is True
|
|
assert (answer["bundle_id"], answer["path"], answer["ref"]) == (
|
|
"tunnel-hauglia",
|
|
path,
|
|
"Krav 12.1",
|
|
)
|
|
assert set(answer) == {
|
|
"declared",
|
|
"bundle_id",
|
|
"path",
|
|
"ref",
|
|
"approach_id",
|
|
"title",
|
|
"req_number",
|
|
"binds",
|
|
}
|
|
assert declared == [
|
|
DeclaredRequirement(
|
|
bundle_id="tunnel-hauglia", path=path, ref="Krav 12.1", approach_id="a1"
|
|
)
|
|
]
|
|
|
|
|
|
def test_a_declaration_naming_an_unknown_base_is_refused() -> None:
|
|
"""(c) The same base index every read rung uses, so no second, laxer resolution exists."""
|
|
tools, opened, declared = _wired()
|
|
opened.append(ToolCall(name="read_file", bundle_id="nope", path="a.md"))
|
|
answer = tools["declare_requirement"].func(
|
|
approach_id="a1", bundle_id="nope", path="a.md", ref="1"
|
|
)
|
|
assert answer["refusal"] == "ExplorationError"
|
|
assert declared == []
|
|
|
|
|
|
def test_the_rung_exists_only_when_both_sinks_are_offered() -> None:
|
|
"""(d) Every pre-P19 call site is byte-identical, and a half-wired one is refused."""
|
|
plain = {t.name for t in navigator_tools((str(_TUNNEL),))}
|
|
assert plain == {"list_bundles", "read_bundle", "read_dir", "read_file"}
|
|
wired, _, _ = _wired()
|
|
assert set(wired) == plain | {"declare_requirement"}
|
|
with pytest.raises(explore.ExplorationError, match="together or not at all"):
|
|
navigator_tools((str(_TUNNEL),), requirements=[])
|
|
with pytest.raises(explore.ExplorationError, match="together or not at all"):
|
|
navigator_tools((str(_TUNNEL),), opened=[])
|
|
|
|
|
|
# ---------------------------------------------------------------------------------------------
|
|
# (e)-(f) the marked hypothesis
|
|
# ---------------------------------------------------------------------------------------------
|
|
|
|
|
|
def _line(**payload: Any) -> str:
|
|
return f"{explore.HYPOTHESIS_MARKER} " + json.dumps({"label": "L", "rationale": "R", **payload})
|
|
|
|
|
|
def test_the_requirement_key_is_never_omitted() -> None:
|
|
"""(e) A marked line that skips the question is the unreadable-marked-line class."""
|
|
with pytest.raises(HypothesisParseError, match="must carry a 'requirement'"):
|
|
explore._parse_hypotheses([_line()], ())
|
|
|
|
|
|
def test_an_explicit_null_is_legal_and_must_say_why() -> None:
|
|
"""(e) "the base holds none" is a FINDING; an unexplained null is a silence."""
|
|
((label, rationale, _base, requirement),) = explore._parse_hypotheses(
|
|
[_line(requirement=None, why_none="the base is a standard, not a price schedule")], ()
|
|
)
|
|
assert (label, rationale, requirement) == ("L", "R", None)
|
|
with pytest.raises(HypothesisParseError, match="why_none"):
|
|
explore._parse_hypotheses([_line(requirement=None)], ())
|
|
|
|
|
|
def test_a_half_named_requirement_is_refused() -> None:
|
|
"""(e) Validation, never repair: a citation that points at nothing is worse than none."""
|
|
with pytest.raises(HypothesisParseError, match="non-empty 'path' and 'ref'"):
|
|
explore._parse_hypotheses([_line(requirement={"path": "a.md"})], ())
|
|
with pytest.raises(HypothesisParseError, match="non-empty 'path' and 'ref'"):
|
|
explore._parse_hypotheses([_line(requirement={"ref": "12.1"})], ())
|
|
|
|
|
|
def test_a_minted_approach_carries_it_and_a_seed_never_acquires_one() -> None:
|
|
"""(f) § C.6 door 1 is a preservation rule."""
|
|
seed = Approach(id="expert-1", label="Seeded", description="the expert's own")
|
|
minted = explore._mint_approaches(
|
|
(seed,), [("Shaped", "in the loop", "", BindingRequirement(path="k/12-1.md", ref="12.1"))]
|
|
)
|
|
assert minted[0] is seed and minted[0].requirement is None
|
|
assert minted[1].requirement == BindingRequirement(path="k/12-1.md", ref="12.1")
|
|
|
|
|
|
# ---------------------------------------------------------------------------------------------
|
|
# (g) A3 — the requirement reaches the proposal
|
|
# ---------------------------------------------------------------------------------------------
|
|
|
|
|
|
def _project() -> Project:
|
|
return Project(id="X", name="Y", description="d", currency="NOK", docs_dir=".", cost_items=())
|
|
|
|
|
|
def test_the_binding_requirement_reaches_the_proposer_verbatim() -> None:
|
|
"""(g) A3, and the byte-identical half is what keeps the golden transcript unchanged."""
|
|
bare = Approach(id="a1", label="L", description="D")
|
|
with_req = bare.model_copy(
|
|
update={"requirement": BindingRequirement(path="R761/12-1/x.md", ref="12.1")}
|
|
)
|
|
without = _build_messages(_project(), "ctx", approach=bare)[0].text
|
|
withit = _build_messages(_project(), "ctx", approach=with_req)[0].text
|
|
assert "Binding requirement" not in without
|
|
assert "Binding requirement: 12.1 (R761/12-1/x.md)" in withit
|
|
# The ONLY difference is the block — an approach without one is byte-identical to before.
|
|
assert (
|
|
withit.replace(
|
|
"Binding requirement: 12.1 (R761/12-1/x.md)\nName that requirement verbatim in 'measure'.\n",
|
|
"",
|
|
)
|
|
== without
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------------------------
|
|
# (h) A4 — the judge
|
|
# ---------------------------------------------------------------------------------------------
|
|
|
|
|
|
def test_a_hit_is_counted_against_this_approachs_fasit_never_the_base() -> None:
|
|
"""(h) The P16 vacuity, one column over: on a 2 756-document base everything is 'in the base'."""
|
|
wanted = {"krav/12-1/a.md", "krav/12-12/b.md"}
|
|
approach = Approach(
|
|
id="a1",
|
|
label="L",
|
|
requirement=BindingRequirement(path="krav/12-1/a.md", ref="12.1"),
|
|
)
|
|
paths, source = stress._attributable(approach, [{"path": "krav/99-9/elsewhere.md"}])
|
|
assert (paths, source) == (("krav/12-1/a.md",), "approach")
|
|
assert set(paths) & wanted
|
|
|
|
# A run-level declaration (one written before declarations carried an ``approach_id``) that is
|
|
# NOT one of this approach's fasit concepts is reported and is not a hit — the base holds it,
|
|
# which is exactly what must not count.
|
|
bare = Approach(id="a2", label="M")
|
|
paths, source = stress._attributable(bare, [{"path": "krav/99-9/elsewhere.md"}])
|
|
assert (paths, source) == (("krav/99-9/elsewhere.md",), "run")
|
|
assert not set(paths) & wanted
|
|
|
|
# And "declared nothing" is a third, distinct finding.
|
|
assert stress._attributable(bare, []) == ((), "absent")
|
|
|
|
|
|
_GOOD = "krav/N1/id-good.md"
|
|
_OTHER = "krav/N1/id-other.md"
|
|
|
|
|
|
def _minibase(root: Path) -> Path:
|
|
base = root / "minibase"
|
|
(base / "krav" / "N1").mkdir(parents=True)
|
|
(base / "index.md").write_text(
|
|
"---\nbundle_id: minibase\n---\n\n- [good](krav/N1/id-good.md)\n"
|
|
"- [other](krav/N1/id-other.md)\n",
|
|
encoding="utf-8",
|
|
)
|
|
(base / _GOOD).write_text(
|
|
'---\ntype: concept\ntitle: "T"\nreq_number: "Krav 1.2.3-4"\n---\n\nBody.\n',
|
|
encoding="utf-8",
|
|
)
|
|
(base / _OTHER).write_text(
|
|
'---\ntype: concept\ntitle: "Other"\nreq_number: "Krav 9.9.9-9"\n---\n\nOther.\n',
|
|
encoding="utf-8",
|
|
)
|
|
return base
|
|
|
|
|
|
def _context_dir(root: Path) -> Path:
|
|
ctx = root / "ctx"
|
|
(ctx / "docs").mkdir(parents=True)
|
|
(ctx / "bundle.txt").write_text("name: minibase\nbundle_id: minibase\n", encoding="utf-8")
|
|
(ctx / "mandate.json").write_text(
|
|
json.dumps(
|
|
{
|
|
"objective": "o",
|
|
"success_criteria": "s",
|
|
"approaches": [
|
|
{
|
|
"id": "a1",
|
|
"label": "L",
|
|
"affected_codes": ["CODE-1"],
|
|
"bundle_id": "minibase",
|
|
}
|
|
],
|
|
}
|
|
),
|
|
encoding="utf-8",
|
|
)
|
|
(ctx / "fasit.json").write_text(
|
|
json.dumps(
|
|
{
|
|
"project_id": "proj",
|
|
"bundle": "minibase",
|
|
"bundle_id": "minibase",
|
|
"must_cite": [
|
|
{
|
|
"approach_id": "a1",
|
|
"rationale": "why",
|
|
"concepts": [{"path": _GOOD, "title": "T", "ref": "Krav 1.2.3-4"}],
|
|
}
|
|
],
|
|
"must_refuse": [],
|
|
"honesty": "synthetic",
|
|
}
|
|
),
|
|
encoding="utf-8",
|
|
)
|
|
return ctx
|
|
|
|
|
|
def _outbox(root: Path, *, declared: str) -> Path:
|
|
out = root / "out"
|
|
out.mkdir(parents=True, exist_ok=True)
|
|
(out / "r1-a1-proposal.json").write_text(
|
|
json.dumps(
|
|
{
|
|
"proposal": {
|
|
"project_id": "proj",
|
|
"measure": "m",
|
|
"affected_items": [{"code": "CODE-1", "quantity": 1.0, "unit_cost": 2.0}],
|
|
"claimed_saving_nok": 1.0,
|
|
"assumptions": {},
|
|
},
|
|
"provenance": {"citations": [], "token_usage": 10},
|
|
}
|
|
),
|
|
encoding="utf-8",
|
|
)
|
|
(out / "r1-a1-outcome.json").write_text(
|
|
json.dumps({"outcome_type": "rejected", "reason": "no"}), encoding="utf-8"
|
|
)
|
|
(out / "r1-debate.json").write_text(
|
|
json.dumps(
|
|
{
|
|
"run_id": "r1",
|
|
"tool_calls": [],
|
|
"requirements": [
|
|
{"bundle_id": "minibase", "path": declared, "ref": "Krav 9.9.9-9"}
|
|
],
|
|
}
|
|
),
|
|
encoding="utf-8",
|
|
)
|
|
return out
|
|
|
|
|
|
def test_a_declaration_in_the_base_but_not_in_the_fasit_is_not_a_hit(tmp_path: Path) -> None:
|
|
"""(h) END-TO-END, and this arm exists because the helper-only one was VACUOUS.
|
|
|
|
MEASURED: mutation A5-(iv) — the judge matching ``concept_names`` instead of this approach's
|
|
own ``wanted`` set — left the WHOLE suite green (1710/5), because the arm above drives
|
|
``_attributable`` while the hit is computed at the call site in ``score_context_set``. The
|
|
declaration here names a document the base really does hold; what it is not is the one the
|
|
fasit asks for, which is the only distinction a 2 756-document corpus leaves standing.
|
|
"""
|
|
base = _minibase(tmp_path)
|
|
ctx = _context_dir(tmp_path)
|
|
|
|
miss = stress.score_context_set(ctx, _outbox(tmp_path, declared=_OTHER), "r1", base)
|
|
assert miss.approaches[0].requirement_declared == (_OTHER,)
|
|
assert miss.approaches[0].requirement_source == "run"
|
|
assert miss.approaches[0].requirement_hit is False
|
|
assert miss.requirements_declared == (_OTHER,)
|
|
|
|
# The positive control: a gate that can only come out false proves nothing either.
|
|
hit = stress.score_context_set(ctx, _outbox(tmp_path, declared=_GOOD), "r1", base)
|
|
assert hit.approaches[0].requirement_hit is True
|
|
|
|
|
|
# ---------------------------------------------------------------------------------------------
|
|
# (i) the artefact
|
|
# ---------------------------------------------------------------------------------------------
|
|
|
|
|
|
def test_the_declaration_reaches_the_debate_artefact(tmp_path: Path) -> None:
|
|
"""(i) A paid run's evidence must survive the process that produced it."""
|
|
from portfolio_optimiser import outbox
|
|
|
|
outbox.write_debate_tools(
|
|
str(tmp_path),
|
|
"r1",
|
|
tool_calls=explore.tool_call_payload(
|
|
[ToolCall(name="read_file", bundle_id="b", path="k/a.md")]
|
|
),
|
|
requirements=explore.requirement_payload(
|
|
[DeclaredRequirement(bundle_id="b", path="k/a.md", ref="12.1", approach_id="a1")]
|
|
),
|
|
)
|
|
payload = json.loads((tmp_path / "r1-debate.json").read_text(encoding="utf-8"))
|
|
assert payload["requirements"] == [
|
|
{"bundle_id": "b", "path": "k/a.md", "ref": "12.1", "approach_id": "a1"}
|
|
]
|
|
# An empty list is the honest positive statement every pre-P19 run makes.
|
|
outbox.write_debate_tools(str(tmp_path), "r2", tool_calls=[])
|
|
assert (
|
|
json.loads((tmp_path / "r2-debate.json").read_text(encoding="utf-8"))["requirements"] == []
|
|
)
|