open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 4 Activity

feat(p19): a direction must NAME the requirement that binds it, and have READ it

Browse source 
Two paid rounds scored 0 of 26 fasit concepts opened -- the same number twice.
P18 closed the navigation side (a listing is a window, an invented path is
refused by name) and it did not move, which makes it a ROLE question: nothing
in the loop ever asked the model to say what requirement binds the direction it
committed to, so opening one was never on the critical path to an answer.

A PREMISE OF THE ORDER WAS FELLED BEFORE ANYTHING WAS BUILT ON IT. A1 places
the demand in _INSTRUCTIONS[HYPOTHESISER_ROLE] alone. Measured: the stress
command sends --mandate and NOT --explore, the two are refused together by
name, and none of the nine round-1/2 outboxes holds a {run_id}-exploration.json
-- the hypothesiser never runs in a stress round, so A3 would have been
unreachable in exactly the paid runs this order commissions.

A2's own sentence resolves it: the refusal goes to the model "som en tur den
kan rette (samme mekanisme som quick_validate's nekt), ikke som en raise" --
and quick_validate IS a tool. declare_requirement therefore lives in
navigator_tools, held by BOTH roles that navigate (the exploration, and since
S2c the debate). It EXISTS only when the caller offers both sinks, which keeps
every pre-P19 call site byte-identical; one sink without the other is refused
at construction. 'opened' is the SAME list ExplorationToolRecorder fills, so
the refusal reads the run's own read trace.

The marked hypothesis carries 'requirement' as a REQUIRED key: omitted is a
hard error, explicit null is legal and needs 'why_none', a half-named one is
refused. A minted approach carries it; a seed never acquires one. The proposer
prompt names it only when the field exists, and the judge counts a hit against
THIS approach's fasit concepts, never against the base.

Load-bearing measured (12 arms), four mutations all red against the whole
suite, green control 1711/5 and demo-transcript.stdout byte-unchanged.
A-iii's predicted signature was FALSIFIED: the golden stays green because the
demo runs without a mandate, so _build_messages' approach branch is never
taken there. A-iv was GREEN first -- the repo's vacuous-gate class, 24th time:
the arm drove _attributable while the hit is computed at the call site.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-15 01:21:47 +02:00
commit c84e8bf6f1
22 changed files with 903 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

207
src/portfolio_optimiser/explore.py
View file

@ -50,7 +50,12 @@ from portfolio_optimiser import okf
from portfolio_optimiser.backends import Profile
from portfolio_optimiser.budget import Budget, BudgetExceeded, BudgetMiddleware, TokenMeter
from portfolio_optimiser.ir import SavingsProposal
from portfolio_optimiser.mandate import OWN_PROPOSAL_ID, Approach, Mandate
from portfolio_optimiser.mandate import (
OWN_PROPOSAL_ID,
Approach,
BindingRequirement,
Mandate,
)
from portfolio_optimiser.retrieval import safe_resolve
from portfolio_optimiser.tracing import exploration_tracer
from portfolio_optimiser.validator import Rejection, validate_proposal
@ -205,10 +210,17 @@ _INSTRUCTIONS: Final = {
),
HYPOTHESISER_ROLE: (
"You shape ONE candidate cost-saving direction at a time from what the navigator found. "
"BEFORE you commit to a direction, name the ONE requirement in the knowledge base that "
"BINDS it: have the navigator find it with read_dir(filter=...) and read it with "
"read_file, then call declare_requirement with the base id, that path and the "
"requirement's own number. A direction with no requirement behind it is a guess. "
"You may call quick_validate to sanity-check a candidate's numbers; its verdict is "
"ADVISORY and is not the project's decision. When you commit to a direction, end your "
f'turn with a line of the form: {HYPOTHESIS_MARKER} {{"label": "<short name>", '
'"rationale": "<why this project, in your own words>"}'
'"rationale": "<why this project, in your own words>", "requirement": '
'{"path": "<the path you read>", "ref": "<the requirement number>"}}. If the base '
'genuinely holds no requirement for this direction, send "requirement": null and '
'"why_none": "<why the base has none>" instead — the field is never omitted.'
),
}
@ -333,6 +345,35 @@ class ToolCall:
path: str
@dataclass(frozen=True)
class DeclaredRequirement:
"""One requirement a navigating role DECLARED as binding, after reading it (P19 DEL A).
Recorded on a CALLER-OWNED sink for the reason ``ToolCall`` is: the declaration is made mid-run
by a tool, and a budget stop after it constructs no result at all — so a returned value would
lose exactly the evidence a paid run was bought for.
"""
bundle_id: str
path: str
ref: str
class RequirementNotRead(ValueError):
"""A role declared a binding requirement it never opened (P19 A2).
**A refusal the model can act on, never a raise that ends the run.** The declaration is a claim
about the corpus, and the cheapest falsifier of it is the run's own read trace: a path that is
not among this run's ``read_file`` calls was not read, whatever the declaration says. Answering
that as a refused TURN — ``quick_validate``'s mechanism, one field over — leaves the model able
to go and read it; raising would end a run over a mistake that costs one tool call to fix.
A ``ValueError``, the ``BundlePathNotFound``/``DimensionScopeRefused`` precedent: the caller is
a model choosing a path, so if it ever escapes a tool body it belongs on the CLI's refusal
tuple and hosting's 400 arm rather than the crash channel.
"""
def _string_argument(arguments: Any, key: str) -> str:
"""One named argument of a call, from either shape ``FunctionInvocationContext`` allows.
@ -415,6 +456,11 @@ class ExplorationTrace:
#: like a successful one on every other field, which is what made the dress rehearsal vacuous
#: by construction and unreadable after the fact.
tool_calls: list[ToolCall] = field(default_factory=list)
#: Every requirement a role DECLARED as binding, in declaration order (P19 DEL A). Beside
#: ``tool_calls`` rather than derived from it: the trace says which documents were opened, this
#: says which one the model committed to as the thing that binds — two different facts, and the
#: second cannot be inferred from the first.
requirements: list[DeclaredRequirement] = field(default_factory=list)
#: Tokens spent so far, refreshed as the loop turns rather than written once at the end. The
#: meter is internal to ``explore``, so this is the only way the artefact can report a spend —
#: and updating it per iteration is what makes it readable for a run a cap cut short, which is
@ -517,6 +563,10 @@ def trace_payload(
for call in trace.quick_validations
],
"tool_calls": tool_call_payload(trace.tool_calls),
# P19 DEL A: what this run declared as binding, beside what it opened. An empty list is an
# honest positive statement — the run declared nothing — which is ``Bundle.skipped``'s
# empty tuple rather than a field that has to be inferred from an absence.
"requirements": requirement_payload(trace.requirements),
}
@ -938,6 +988,7 @@ def _index_excerpt(body: str) -> tuple[str, bool]:
#: reachable ``ExplorationError`` is ``_resolve_bundle``'s unknown-base refusal.
_RETURNABLE_REFUSALS: Final = (
ExplorationError,
RequirementNotRead,
okf.BundleIdMismatch,
okf.BundlePathNotFound,
okf.DocumentPathRefused,
@ -1005,9 +1056,14 @@ def _refused_text(exc: Exception) -> str:
def navigator_tools(
bundle_dirs: Sequence[str], *, dimension: str | None = None
bundle_dirs: Sequence[str],
*,
dimension: str | None = None,
opened: list[ToolCall] | None = None,
requirements: list[DeclaredRequirement] | None = None,
) -> list[FunctionTool]:
"""The navigator's three tools: survey the catalogue, open one base, read one document.
"""The navigator's tools: survey the catalogue, open one base, read one document — and, when
the caller offers the two sinks, DECLARE the requirement that binds a direction.
Progressive disclosure, not stuffing (målbilde §2/§4): ``list_bundles`` never returns content,
only what each base IS — and crucially whether it ships a ``cost-baseline.json``, because a base
@ -1052,7 +1108,21 @@ def navigator_tools(
hides a foreign-dimension document while ``read_file`` still serves it by path is a filter in
name only — a model-chosen path is untrusted input, so the gate belongs where the bytes leave.
``None`` (the exploration's own call) admits everything, byte-identical to before.
**``declare_requirement`` exists only when the caller passes BOTH sinks** (P19 DEL A), and that
is what keeps every other call site byte-identical — including the tool-set assertions three
older gates make. ``opened`` is the SAME list ``ExplorationToolRecorder`` appends to (an alias,
never a copy — the ``_drive`` rule), because the refusal this tool exists for is answered by
the run's own read trace and a second record of it would be free to disagree with the first.
Passing one without the other is refused at construction: a log that cannot see what was opened
would accept every declaration, which is the vacuous-gate class.
"""
if (opened is None) != (requirements is None):
raise ExplorationError(
"navigator_tools takes 'opened' and 'requirements' together or not at all: a "
"requirement sink with no read trace could not refuse an undeclared read, and a read "
"trace with no sink would record nothing"
)
index = _bundle_index(bundle_dirs)
@tool(
@ -1239,7 +1309,52 @@ def navigator_tools(
)
return resolved.read_text(encoding="utf-8")
return [list_bundles, read_bundle, read_dir, read_file]
@tool(
name="declare_requirement",
description=(
"Declare the ONE requirement in a knowledge base that BINDS the direction you are "
"about to commit to, by base id, the bundle-relative path read_file gave you, and the "
"requirement's own number as its frontmatter states it. You must have READ the "
"document with read_file first: a declaration naming a path this run never opened is "
"refused, and reading it is the correction. Use read_dir with a 'filter' word to find "
"it, read_file to read it, then declare it."
),
)
def declare_requirement(bundle_id: str, path: str, ref: str) -> dict[str, Any]:
try:
return _declare_requirement(bundle_id, path, ref)
except _RETURNABLE_REFUSALS as exc:
return _refused_mapping(exc)
def _declare_requirement(bundle_id: str, path: str, ref: str) -> dict[str, Any]:
assert opened is not None and requirements is not None # the constructor guard above
# The base is resolved by the SAME index every read rung uses, so an unknown base is
# refused here exactly as it is there rather than being accepted into the record.
_resolve_bundle(index, bundle_id)
read_paths = [call.path for call in opened if call.name == "read_file" and call.path]
if path not in read_paths:
raise RequirementNotRead(
f"{path!r}; this run has opened {len(read_paths)} document(s) with read_file, and "
"this is not one of them. Read it first — a requirement nobody read cannot bind a "
"direction"
)
requirements.append(DeclaredRequirement(bundle_id=bundle_id, path=path, ref=ref))
return {"declared": True, "bundle_id": bundle_id, "path": path, "ref": ref}
tools = [list_bundles, read_bundle, read_dir, read_file]
if requirements is not None:
tools.append(declare_requirement)
return tools
def requirement_payload(declared: Sequence[DeclaredRequirement]) -> list[dict[str, Any]]:
"""The ONE rendering of declared requirements into plain data, in DECLARATION order.
Two artefacts carry it — ``{run_id}-exploration.json`` and ``{run_id}-debate.json`` — and two
copies of "what a declaration looks like" would drift into two answers about one run, which is
the kø-(p) defect landing in exactly the files an operator reads after a paid run.
"""
return [{"bundle_id": d.bundle_id, "path": d.path, "ref": d.ref} for d in declared]
def _refused(
@ -1369,6 +1484,8 @@ def fresh_exploration_workflow(
bundle_dirs: Sequence[str] = (),
middleware: Sequence[Any] | None = None,
quick_validate_sink: list[QuickValidation] | None = None,
tool_call_sink: list[ToolCall] | None = None,
requirement_sink: list[DeclaredRequirement] | None = None,
checkpoint_dir: str | None = None,
) -> Any:
"""Build a FRESH Magentic workflow with fresh agents and fresh clients (mirrors
@ -1392,9 +1509,18 @@ def fresh_exploration_workflow(
token guarantee: agent-level ``ChatMiddleware`` does fire on the manager's own calls (measured
A1), and the manager is the most talkative participant in the loop.
"""
# P19 DEL A: the declaration tool reaches BOTH roles, and that is a measurement rather than
# generosity. The instruction that asks for a binding requirement is the HYPOTHESISER's — it is
# the role that commits to a direction — while the ``read_file`` calls the refusal checks are
# the NAVIGATOR's. Giving it to the navigator alone would leave the committing role unable to
# state its own commitment; to the hypothesiser alone, unable to declare what its partner read.
navigator = list(
navigator_tools(bundle_dirs, opened=tool_call_sink, requirements=requirement_sink)
)
hypothesiser_tools: list[Any] = [quick_validate_tool(bundle_dirs, sink=quick_validate_sink)]
hypothesiser_tools += [t for t in navigator if getattr(t, "name", "") == "declare_requirement"]
tools_by_role: dict[str, list[Any]] = {
NAVIGATOR_ROLE: list(navigator_tools(bundle_dirs)),
NAVIGATOR_ROLE: navigator,
HYPOTHESISER_ROLE: hypothesiser_tools,
}
participants = [
@ -1583,13 +1709,45 @@ def _resolve_hypothesis_bundle(raw: str, bundle_ids: Sequence[str]) -> str:
)
def _requirement_of(data: Mapping[str, Any], line: str) -> BindingRequirement | None:
"""The marked hypothesis's binding requirement, or ``None`` when the base genuinely has none.
**The field is never OMITTED** (P19 A1): a missing key is a hard error of the same class as an
unreadable marked line, because the marker is what makes fail-closed affordable — the model
committed to a direction, and "which requirement binds it" is part of that commitment rather
than an optional extra. An EXPLICIT ``null`` is legal and needs ``why_none``: a base that holds
no requirement for a direction is a finding worth stating, and one stated without a reason is
indistinguishable from the model having skipped the question.
"""
if "requirement" not in data:
raise HypothesisParseError(
"a marked hypothesis must carry a 'requirement' — either "
'{"path": ..., "ref": ...} for the document that binds it, or null together with '
f"'why_none'. A direction with no requirement behind it is a guess; got: {line}"
)
raw = data["requirement"]
if raw is None:
if not data.get("why_none"):
raise HypothesisParseError(
"a marked hypothesis with 'requirement': null must say 'why_none' — the base "
f"holding no requirement is a finding, and an unexplained null is a silence: {line}"
)
return None
if not isinstance(raw, dict) or not raw.get("path") or not raw.get("ref"):
raise HypothesisParseError(
"a marked hypothesis's 'requirement' needs a non-empty 'path' and 'ref'; a half-named "
f"requirement reads as a citation and points at nothing: {line}"
)
return BindingRequirement(path=str(raw["path"]), ref=str(raw["ref"]))
def _parse_hypotheses(
texts: Sequence[str], bundle_ids: Sequence[str]
) -> list[tuple[str, str, str]]:
"""Every marked ``(label, rationale, bundle_id)`` triple the hypothesiser committed to, in turn
order. The base is resolved HERE rather than at minting time so an unroutable claim is refused
while the line that made it is still in hand for the error message."""
found: list[tuple[str, str, str]] = []
) -> list[tuple[str, str, str, BindingRequirement | None]]:
"""Every marked ``(label, rationale, bundle_id, requirement)`` the hypothesiser committed to,
in turn order. The base is resolved HERE rather than at minting time so an unroutable claim is
refused while the line that made it is still in hand for the error message."""
found: list[tuple[str, str, str, BindingRequirement | None]] = []
for text in texts:
for line in text.splitlines():
stripped = line.strip()
@ -1613,6 +1771,7 @@ def _parse_hypotheses(
str(data["label"]),
str(data["rationale"]),
_resolve_hypothesis_bundle(str(data.get("bundle_id") or ""), bundle_ids),
_requirement_of(data, stripped),
)
)
return found
@ -1649,7 +1808,8 @@ def refuse_unroutable_seeds(seeds: Sequence[Approach], bundle_ids: Sequence[str]
def _mint_approaches(
seeds: Sequence[Approach], discovered: Sequence[tuple[str, str, str]]
seeds: Sequence[Approach],
discovered: Sequence[tuple[str, str, str, BindingRequirement | None]],
) -> tuple[Approach, ...]:
"""Seeds FIRST, untouched, then one approach per discovered direction.
@ -1662,7 +1822,7 @@ def _mint_approaches(
taken = {approach.id for approach in seeds}
minted: list[Approach] = list(seeds)
counter = 0
for label, rationale, bundle_id in discovered:
for label, rationale, bundle_id, requirement in discovered:
counter += 1
candidate = f"hypothesis-{counter}"
while candidate in taken or candidate == OWN_PROPOSAL_ID:
@ -1675,8 +1835,17 @@ def _mint_approaches(
#
# ``bundle_id`` is stamped on a MINTED approach because there is nothing here to preserve —
# the opposite call from the seeds above, which pass through untouched (§ C.6 door 1).
# ``requirement`` is stamped on a MINTED approach for the same reason ``bundle_id`` is:
# there is nothing here to preserve. A SEED passes through untouched (§ C.6 door 1) — an
# expert who named no requirement is not to be given one on their behalf.
minted.append(
Approach(id=candidate, label=label, description=rationale, bundle_id=bundle_id)
Approach(
id=candidate,
label=label,
description=rationale,
bundle_id=bundle_id,
requirement=requirement,
)
)
return tuple(minted)
@ -1801,6 +1970,11 @@ async def explore(
bundle_dirs=bundle_dirs,
middleware=[BudgetMiddleware(meter), ExplorationToolRecorder(trace.tool_calls)],
quick_validate_sink=trace.quick_validations,
# ALIASES of the trace's own lists, never copies (the ``_drive`` rule): the refusal reads
# the same read trace the recorder writes, so the two can never disagree about what this
# run opened.
tool_call_sink=trace.tool_calls,
requirement_sink=trace.requirements,
checkpoint_dir=checkpoint_dir,
)
@ -2093,6 +2267,11 @@ async def resume_exploration(
bundle_dirs=parked.bundle_dirs,
middleware=[BudgetMiddleware(meter), ExplorationToolRecorder(trace.tool_calls)],
quick_validate_sink=trace.quick_validations,
# ALIASES of the trace's own lists, never copies (the ``_drive`` rule): the refusal reads
# the same read trace the recorder writes, so the two can never disagree about what this
# run opened.
tool_call_sink=trace.tool_calls,
requirement_sink=trace.requirements,
checkpoint_dir=checkpoint_dir,
)

11
src/portfolio_optimiser/generate.py
View file

@ -327,6 +327,17 @@ def _build_messages(
)
if approach.description:
head += f"Why the expert wants it evaluated: {approach.description}\n"
# P19 A3: the ONE requirement of the knowledge base that binds this direction, when one was
# declared. The line exists only when the field does, so every prompt written before today
# — the demo's included, which is what keeps the golden transcript byte-identical — is
# unchanged by construction. ``ref`` leads because that is what the model is asked to
# restate; the path follows so the claim can be checked against what the run opened.
if approach.requirement is not None:
head += (
f"Binding requirement: {approach.requirement.ref} "
f"({approach.requirement.path})\n"
"Name that requirement verbatim in 'measure'.\n"
)
prompt = (
f"{head}"
f"Project: {project.id} - {project.name}\n"

34
src/portfolio_optimiser/mandate.py
View file

@ -43,6 +43,31 @@ from portfolio_optimiser.ir import AffectedItem, CostBaseline, SavingsProposal
OWN_PROPOSAL_ID = "own-proposal"
class BindingRequirement(BaseModel):
"""The ONE document in the knowledge base that BINDS a direction (P19 DEL A).
Measured over two paid stress rounds (P16 § 3, P18 § 1): **0 of 26** fasit concepts were ever
opened, in both rounds, while every run still produced proposals — so a direction could be
committed to, quantified and validated without a single requirement of the corpus having been
read. P18 closed the navigation side of that (a window, a named refusal for an invented path)
and the number did not move, which is what made it a ROLE question rather than a ladder one:
nothing in the loop ever asked the model to name what binds it.
``path`` is bundle-relative exactly as a listing gave it — the same string ``read_file`` takes,
so it can be checked against what the run actually opened without a second normalisation.
``ref`` is the requirement's OWN number as the document's frontmatter states it, never a
paraphrase: it is what a reader searches the corpus with, and what the judge matches the fasit
on.
Both are required with ``min_length=1``. A half-declared requirement would be worse than none:
it reads as a citation and points at nothing, which is the fabricated-provenance class
``write_concept_file`` refuses and ``RunFailure`` names in its own docstring.
"""
path: str = Field(min_length=1)
ref: str = Field(min_length=1)
class Approach(BaseModel):
"""One approach a domain expert wants evaluated for a project.
@ -84,6 +109,15 @@ class Approach(BaseModel):
#: The field is a ROUTING key, never a claim about content. It says which pipeline the approach
#: must be evaluated in, and ``route_by_bundle`` is the one place that reads it.
bundle_id: str = ""
#: The ONE requirement of the knowledge base that binds this direction (P19 DEL A).
#:
#: DEFAULTS to ``None``, which keeps every mandate written before today valid and dispatchable
#: unchanged — the ``bundle_id`` precedent, and the honest reading of a commission whose author
#: named no requirement. ``None`` is therefore "none stated", never "none exists": the loop
#: that MINTS an approach must say which of the two it means (``why_none`` on the hypothesis
#: line), because a direction the base has no requirement for is a finding, while one nobody
#: looked for is a silence.
requirement: BindingRequirement | None = None
class Mandate(BaseModel):

15
src/portfolio_optimiser/outbox.py
View file

@ -194,8 +194,10 @@ def write_debate_tools(
run_id: str,
*,
tool_calls: Sequence[Mapping[str, Any]],
requirements: Sequence[Mapping[str, Any]] = (),
) -> Path:
"""Write ``{run_id}-debate.json`` — WHICH documents the debate opened, in call order (S2c).
"""Write ``{run_id}-debate.json`` — WHICH documents the debate opened, in call order (S2c), and
WHICH requirement it declared as binding (P19 DEL A).
The sibling of ``write_exploration`` one phase over. Since S2c the debate navigates the
knowledge base instead of being handed it whole, so "what did this run actually read" is a
@ -213,7 +215,16 @@ def write_debate_tools(
directory.mkdir(parents=True, exist_ok=True)
path = directory / f"{run_id}-debate.json"
path.write_text(
_dump({"run_id": run_id, "tool_calls": [dict(call) for call in tool_calls]}),
_dump(
{
"run_id": run_id,
"tool_calls": [dict(call) for call in tool_calls],
# DEFAULTS to empty for the same reason ``Bundle.skipped`` does: "this debate
# declared nothing" is an honest positive statement, and it is the one every run
# written before today makes.
"requirements": [dict(r) for r in requirements],
}
),
encoding="utf-8",
)
return path

41
src/portfolio_optimiser/run.py
View file

@ -64,6 +64,7 @@ from portfolio_optimiser.explore import (
ExplorationResult,
ExplorationToolRecorder,
ExplorationTrace,
DeclaredRequirement,
ParkedStateError,
PlanReviewDecision,
PlanReviewParked,
@ -75,6 +76,7 @@ from portfolio_optimiser.explore import (
parked_notice,
parked_payload,
navigator_tools,
requirement_payload,
resume_exploration,
terminal_plan_reviewer,
tool_call_payload,
@ -607,7 +609,11 @@ def _bundle_pointer(bundle: okf.Bundle, bundle_id: str, *, dimension: str | None
"A listing is a WINDOW: it reports 'total' for the level and gives you 'limit' entries "
"from 'offset'. When 'total' is large, do not page through it — narrow it: "
f"read_dir({bundle_id!r}, path, filter='<word>') answers with the entries whose title, "
"requirement number or path contains that word, and reports 'total_matches'."
"requirement number or path contains that word, and reports 'total_matches'.\n"
"Before you settle on a measure, name the ONE requirement of this base that BINDS it: "
"find it with a filter, read it with read_file, then call "
f"declare_requirement({bundle_id!r}, path, ref) with the requirement's own number. A "
"declaration naming a document this run never opened is refused; reading it is the fix."
)
@ -1116,6 +1122,21 @@ async def run_project(
# gate's share rule needs, and composing them here — where the base is already walked — is what
# keeps them from being a second, drifting reconstruction (kø-(p)).
bundle_grounding: tuple[str, ...] = ()
# S2c: a CALLER-OWNED sink for what the debate opens (the ``parse_failures``/``ExplorationTrace``
# shape). A returned value would be lost on exactly the run that most needs the evidence — a
# budget stop mid-debate raises out of ``debate.run`` and constructs no ``RunResult`` at all.
# ``ExplorationToolRecorder`` is REUSED rather than re-implemented: it is already the recorder
# for in-process navigator calls, ordered and un-deduplicated, which is exactly the question
# here too ("did this run open anything, and in what sequence"). Its sibling
# ``mcp_tools.ToolCallRecorder`` stays what it is — a sorted, de-duplicated EGRESS claim.
#
# BOUND HERE, above the fork, because the bundle arm hands both lists to ``navigator_tools``:
# the declaration rung refuses against the very trace the recorder writes, and a second list
# would be free to disagree with it about what this run opened (kø-(p)).
debate_tool_calls: list[ToolCall] = []
#: P19 DEL A: which requirement the debate declared as binding, in declaration order.
debate_requirements: list[DeclaredRequirement] = []
if bundle_dir is not None:
bundle = okf.navigate_bundle(bundle_dir)
bundle_grounding = tuple(
@ -1192,7 +1213,17 @@ async def run_project(
# both rungs (``navigator_tools``' own gate), because that is now where the bytes
# leave. Under a payload the SAME two gates are re-raised by
# ``prepass.verify_against_bundle`` on the mounted documents instead.
debate_tools = list(navigator_tools([bundle_dir], dimension=dimension_id))
# P19 DEL A: the debate gets the declaration rung too, and the sinks are what create
# it. ``debate_tool_calls`` is bound below — this list is the SAME one
# ``ExplorationToolRecorder`` fills, so the refusal reads the run's own read trace.
debate_tools = list(
navigator_tools(
[bundle_dir],
dimension=dimension_id,
opened=debate_tool_calls,
requirements=debate_requirements,
)
)
# What the navigation could NOT reach, taken from the run's ONE walk. The road path below
# navigates no bundle at all, so its empty tuple is literally true rather than a stand-in.
skipped_links: tuple[okf.SkippedLink, ...] = bundle.skipped
@ -1266,7 +1297,6 @@ async def run_project(
# for in-process navigator calls, ordered and un-deduplicated, which is exactly the question
# here too ("did this run open anything, and in what sequence"). Its sibling
# ``mcp_tools.ToolCallRecorder`` stays what it is — a sorted, de-duplicated EGRESS claim.
debate_tool_calls: list[ToolCall] = []
debate_middleware: list[Any] = [budget_mw, ExplorationToolRecorder(debate_tool_calls)]
if call_recorder is not None:
debate_middleware.append(call_recorder)
@ -1343,7 +1373,10 @@ async def run_project(
declaration=prepass.declaration_payload(prepass_declaration),
)
outbox.write_debate_tools(
outbox_dir, run_id, tool_calls=tool_call_payload(debate_tool_calls)
outbox_dir,
run_id,
tool_calls=tool_call_payload(debate_tool_calls),
requirements=requirement_payload(debate_requirements),
)
# F1: the candidate must derive from the DEBATE. Feed the proposer's converged output into
# generation (retrieval context is the last-resort fallback only). The checker's verdict

12
src/portfolio_optimiser/simulation.py
View file

@ -745,7 +745,17 @@ def scripted_exploration_factory(
existing ``_proposer_reply`` / ``_CHECKER_APPROVE`` scaffolding, so nothing about the debate
changes."""
pipeline = scripted_factory({"proposer": _proposer_reply, "checker": _CHECKER_APPROVE}, sink)
hypothesis_line = f"{HYPOTHESIS_MARKER} " + json.dumps({"label": label, "rationale": rationale})
hypothesis_line = f"{HYPOTHESIS_MARKER} " + json.dumps(
# P19 A1: ``requirement`` is a required key. The demo's scripted manager routes to the
# hypothesiser without any document having been read, so the explicit-null branch is the
# only honest one here — and it is what keeps the golden transcript byte-identical.
{
"label": label,
"rationale": rationale,
"requirement": None,
"why_none": "scripted demo: no document was read",
}
)
def factory(role: str) -> BaseChatClient:
if role == MANAGER_ROLE:

54
src/portfolio_optimiser/stress.py
View file

@ -69,6 +69,7 @@ import argparse
import json
import os
import sys
from collections.abc import Sequence
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any
@ -110,6 +111,14 @@ class ApproachVerdict:
named_in_snippet: bool
#: (c) - attributable hallucinations, ``citation:<file>`` / ``code:<code>``.
hallucinations: tuple[str, ...]
#: P19 A4 - the binding requirement this row can be attributed, and whether it is one of the
#: fasit's own concepts for this approach. ``requirement_source`` says WHICH of the two places
#: it came from, because they are different claims: ``approach`` is per-approach by
#: construction (the mandate carries it), while ``run`` is the DEBATE's declaration, which is
#: written once per run and therefore cannot be attributed to one approach on its own.
requirement_declared: tuple[str, ...]
requirement_source: str # "approach" | "run" | "absent"
requirement_hit: bool
ferdig: bool
@ -133,6 +142,11 @@ class ContextSetVerdict:
must_refuse: tuple[RefusalVerdict, ...]
#: Run-level: read paths the base does not carry (guessed by the navigator).
hallucinated_reads: tuple[str, ...]
#: P19 A4: every requirement the RUN declared as binding, in declaration order, with the
#: denominator every other field here carries. Reported even when empty - "the run declared
#: none" is the measurement, and a missing field would be indistinguishable from a judge that
#: did not look.
requirements_declared: tuple[str, ...]
tool_calls_seen: int
citations_seen: int
approach_rows_seen: int
@ -171,6 +185,24 @@ def _inside(base: Path, raw: str) -> Path | None:
return resolved if resolved == root or root in resolved.parents else None
def _attributable(approach: Any, declared: Sequence[str]) -> tuple[tuple[str, ...], str]:
"""Which declared requirement paths this approach may be judged on, and where they came from.
The approach's OWN requirement wins when it has one: the mandate carries it per approach, so
it is unambiguous by construction. Otherwise the RUN's declarations are attributable - the
debate declares once for the whole run, so the row says ``run`` rather than pretending the
declaration was made about it. ``absent`` is the third value and is not the same as "declared
nothing that matched": a run that declared nothing is a different finding from one that
declared the wrong document."""
own = getattr(approach, "requirement", None)
path = "" if own is None else str(getattr(own, "path", "") or "")
if path:
return (path,), "approach"
if declared:
return tuple(declared), "run"
return (), "absent"
def score_context_set(
context_dir: str | Path,
outbox_dir: str | Path,
@ -206,6 +238,18 @@ def score_context_set(
)
opened_paths = {c.get("path", "") for c in tool_calls if c.get("name") == "read_file"}
opened_paths.discard("")
# P19 A4: WHERE the declarations live was measured, not assumed. A ``--mandate`` run (which is
# what every stress round has been) writes no ``{run_id}-exploration.json`` at all - the
# hypothesiser never runs - so the debate artefact is the only one that can carry them there.
# Both are read, because an ``--explore`` run carries them in the other.
declared_paths: list[str] = []
for artefact in (debate, outbox / f"{run_id}-exploration.json"):
if artefact.is_file():
declared_paths += [
str(r.get("path", ""))
for r in _read_json(artefact).get("requirements", [])
if r.get("path")
]
hallucinated_reads: list[str] = []
for call in tool_calls:
raw = str(call.get("path", ""))
@ -245,6 +289,9 @@ def score_context_set(
named_in_measure=False,
named_in_snippet=False,
hallucinations=(),
requirement_declared=_attributable(approach, declared_paths)[0],
requirement_source=_attributable(approach, declared_paths)[1],
requirement_hit=bool(set(_attributable(approach, declared_paths)[0]) & wanted),
ferdig=False,
)
)
@ -288,6 +335,9 @@ def score_context_set(
# snippets and gave this row ``named`` without the model having said anything.
named_in_snippet = scope == "narrowed" and any(m in snippets for m in marks)
attributable, requirement_source = _attributable(approach, declared_paths)
requirement_hit = bool(set(attributable) & wanted)
halluc = [f"citation:{f}" for f in sorted(cited_files - concept_names)]
allowed = set(approach.affected_codes) | baseline_codes
halluc += [
@ -310,6 +360,9 @@ def score_context_set(
named_in_measure=named_in_measure,
named_in_snippet=named_in_snippet,
hallucinations=tuple(halluc),
requirement_declared=attributable,
requirement_source=requirement_source,
requirement_hit=requirement_hit,
ferdig=(
grounded
and (named_in_measure or named_in_snippet)
@ -355,6 +408,7 @@ def score_context_set(
approaches=tuple(rows),
must_refuse=tuple(refusals),
hallucinated_reads=tuple(hallucinated_reads),
requirements_declared=tuple(declared_paths),
tool_calls_seen=len(tool_calls),
citations_seen=citations_seen,
approach_rows_seen=rows_seen,