open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(p19): a direction must NAME the requirement that binds it, and have READ it

Browse source 
Two paid rounds scored 0 of 26 fasit concepts opened -- the same number twice.
P18 closed the navigation side (a listing is a window, an invented path is
refused by name) and it did not move, which makes it a ROLE question: nothing
in the loop ever asked the model to say what requirement binds the direction it
committed to, so opening one was never on the critical path to an answer.

A PREMISE OF THE ORDER WAS FELLED BEFORE ANYTHING WAS BUILT ON IT. A1 places
the demand in _INSTRUCTIONS[HYPOTHESISER_ROLE] alone. Measured: the stress
command sends --mandate and NOT --explore, the two are refused together by
name, and none of the nine round-1/2 outboxes holds a {run_id}-exploration.json
-- the hypothesiser never runs in a stress round, so A3 would have been
unreachable in exactly the paid runs this order commissions.

A2's own sentence resolves it: the refusal goes to the model "som en tur den
kan rette (samme mekanisme som quick_validate's nekt), ikke som en raise" --
and quick_validate IS a tool. declare_requirement therefore lives in
navigator_tools, held by BOTH roles that navigate (the exploration, and since
S2c the debate). It EXISTS only when the caller offers both sinks, which keeps
every pre-P19 call site byte-identical; one sink without the other is refused
at construction. 'opened' is the SAME list ExplorationToolRecorder fills, so
the refusal reads the run's own read trace.

The marked hypothesis carries 'requirement' as a REQUIRED key: omitted is a
hard error, explicit null is legal and needs 'why_none', a half-named one is
refused. A minted approach carries it; a seed never acquires one. The proposer
prompt names it only when the field exists, and the judge counts a hit against
THIS approach's fasit concepts, never against the base.

Load-bearing measured (12 arms), four mutations all red against the whole
suite, green control 1711/5 and demo-transcript.stdout byte-unchanged.
A-iii's predicted signature was FALSIFIED: the golden stays green because the
demo runs without a mandate, so _build_messages' approach branch is never
taken there. A-iv was GREEN first -- the repo's vacuous-gate class, 24th time:
the arm drove _attributable while the hit is computed at the call site.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-15 01:21:47 +02:00
commit c84e8bf6f1
22 changed files with 903 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

207
src/portfolio_optimiser/explore.py
View file

@ -50,7 +50,12 @@ from portfolio_optimiser import okf
from portfolio_optimiser.backends import Profile
from portfolio_optimiser.budget import Budget, BudgetExceeded, BudgetMiddleware, TokenMeter
from portfolio_optimiser.ir import SavingsProposal
from portfolio_optimiser.mandate import OWN_PROPOSAL_ID, Approach, Mandate
from portfolio_optimiser.mandate import (
OWN_PROPOSAL_ID,
Approach,
BindingRequirement,
Mandate,
)
from portfolio_optimiser.retrieval import safe_resolve
from portfolio_optimiser.tracing import exploration_tracer
from portfolio_optimiser.validator import Rejection, validate_proposal
@ -205,10 +210,17 @@ _INSTRUCTIONS: Final = {
),
HYPOTHESISER_ROLE: (
"You shape ONE candidate cost-saving direction at a time from what the navigator found. "
"BEFORE you commit to a direction, name the ONE requirement in the knowledge base that "
"BINDS it: have the navigator find it with read_dir(filter=...) and read it with "
"read_file, then call declare_requirement with the base id, that path and the "
"requirement's own number. A direction with no requirement behind it is a guess. "
"You may call quick_validate to sanity-check a candidate's numbers; its verdict is "
"ADVISORY and is not the project's decision. When you commit to a direction, end your "
f'turn with a line of the form: {HYPOTHESIS_MARKER} {{"label": "<short name>", '
'"rationale": "<why this project, in your own words>"}'
'"rationale": "<why this project, in your own words>", "requirement": '
'{"path": "<the path you read>", "ref": "<the requirement number>"}}. If the base '
'genuinely holds no requirement for this direction, send "requirement": null and '
'"why_none": "<why the base has none>" instead — the field is never omitted.'
),
}
@ -333,6 +345,35 @@ class ToolCall:
path: str
@dataclass(frozen=True)
class DeclaredRequirement:
"""One requirement a navigating role DECLARED as binding, after reading it (P19 DEL A).
Recorded on a CALLER-OWNED sink for the reason ``ToolCall`` is: the declaration is made mid-run
by a tool, and a budget stop after it constructs no result at all — so a returned value would
lose exactly the evidence a paid run was bought for.
"""
bundle_id: str
path: str
ref: str
class RequirementNotRead(ValueError):
"""A role declared a binding requirement it never opened (P19 A2).
**A refusal the model can act on, never a raise that ends the run.** The declaration is a claim
about the corpus, and the cheapest falsifier of it is the run's own read trace: a path that is
not among this run's ``read_file`` calls was not read, whatever the declaration says. Answering
that as a refused TURN — ``quick_validate``'s mechanism, one field over — leaves the model able
to go and read it; raising would end a run over a mistake that costs one tool call to fix.
A ``ValueError``, the ``BundlePathNotFound``/``DimensionScopeRefused`` precedent: the caller is
a model choosing a path, so if it ever escapes a tool body it belongs on the CLI's refusal
tuple and hosting's 400 arm rather than the crash channel.
"""
def _string_argument(arguments: Any, key: str) -> str:
"""One named argument of a call, from either shape ``FunctionInvocationContext`` allows.
@ -415,6 +456,11 @@ class ExplorationTrace:
#: like a successful one on every other field, which is what made the dress rehearsal vacuous
#: by construction and unreadable after the fact.
tool_calls: list[ToolCall] = field(default_factory=list)
#: Every requirement a role DECLARED as binding, in declaration order (P19 DEL A). Beside
#: ``tool_calls`` rather than derived from it: the trace says which documents were opened, this
#: says which one the model committed to as the thing that binds — two different facts, and the
#: second cannot be inferred from the first.
requirements: list[DeclaredRequirement] = field(default_factory=list)
#: Tokens spent so far, refreshed as the loop turns rather than written once at the end. The
#: meter is internal to ``explore``, so this is the only way the artefact can report a spend —
#: and updating it per iteration is what makes it readable for a run a cap cut short, which is
@ -517,6 +563,10 @@ def trace_payload(
for call in trace.quick_validations
],
"tool_calls": tool_call_payload(trace.tool_calls),
# P19 DEL A: what this run declared as binding, beside what it opened. An empty list is an
# honest positive statement — the run declared nothing — which is ``Bundle.skipped``'s
# empty tuple rather than a field that has to be inferred from an absence.
"requirements": requirement_payload(trace.requirements),
}
@ -938,6 +988,7 @@ def _index_excerpt(body: str) -> tuple[str, bool]:
#: reachable ``ExplorationError`` is ``_resolve_bundle``'s unknown-base refusal.
_RETURNABLE_REFUSALS: Final = (
ExplorationError,
RequirementNotRead,
okf.BundleIdMismatch,
okf.BundlePathNotFound,
okf.DocumentPathRefused,
@ -1005,9 +1056,14 @@ def _refused_text(exc: Exception) -> str:
def navigator_tools(
bundle_dirs: Sequence[str], *, dimension: str | None = None
bundle_dirs: Sequence[str],
*,
dimension: str | None = None,
opened: list[ToolCall] | None = None,
requirements: list[DeclaredRequirement] | None = None,
) -> list[FunctionTool]:
"""The navigator's three tools: survey the catalogue, open one base, read one document.
"""The navigator's tools: survey the catalogue, open one base, read one document — and, when
the caller offers the two sinks, DECLARE the requirement that binds a direction.
Progressive disclosure, not stuffing (målbilde §2/§4): ``list_bundles`` never returns content,
only what each base IS — and crucially whether it ships a ``cost-baseline.json``, because a base
@ -1052,7 +1108,21 @@ def navigator_tools(
hides a foreign-dimension document while ``read_file`` still serves it by path is a filter in
name only — a model-chosen path is untrusted input, so the gate belongs where the bytes leave.
``None`` (the exploration's own call) admits everything, byte-identical to before.
**``declare_requirement`` exists only when the caller passes BOTH sinks** (P19 DEL A), and that
is what keeps every other call site byte-identical — including the tool-set assertions three
older gates make. ``opened`` is the SAME list ``ExplorationToolRecorder`` appends to (an alias,
never a copy — the ``_drive`` rule), because the refusal this tool exists for is answered by
the run's own read trace and a second record of it would be free to disagree with the first.
Passing one without the other is refused at construction: a log that cannot see what was opened
would accept every declaration, which is the vacuous-gate class.
"""
if (opened is None) != (requirements is None):
raise ExplorationError(
"navigator_tools takes 'opened' and 'requirements' together or not at all: a "
"requirement sink with no read trace could not refuse an undeclared read, and a read "
"trace with no sink would record nothing"
)
index = _bundle_index(bundle_dirs)
@tool(
@ -1239,7 +1309,52 @@ def navigator_tools(
)
return resolved.read_text(encoding="utf-8")
return [list_bundles, read_bundle, read_dir, read_file]
@tool(
name="declare_requirement",
description=(
"Declare the ONE requirement in a knowledge base that BINDS the direction you are "
"about to commit to, by base id, the bundle-relative path read_file gave you, and the "
"requirement's own number as its frontmatter states it. You must have READ the "
"document with read_file first: a declaration naming a path this run never opened is "
"refused, and reading it is the correction. Use read_dir with a 'filter' word to find "
"it, read_file to read it, then declare it."
),
)
def declare_requirement(bundle_id: str, path: str, ref: str) -> dict[str, Any]:
try:
return _declare_requirement(bundle_id, path, ref)
except _RETURNABLE_REFUSALS as exc:
return _refused_mapping(exc)
def _declare_requirement(bundle_id: str, path: str, ref: str) -> dict[str, Any]:
assert opened is not None and requirements is not None # the constructor guard above
# The base is resolved by the SAME index every read rung uses, so an unknown base is
# refused here exactly as it is there rather than being accepted into the record.
_resolve_bundle(index, bundle_id)
read_paths = [call.path for call in opened if call.name == "read_file" and call.path]
if path not in read_paths:
raise RequirementNotRead(
f"{path!r}; this run has opened {len(read_paths)} document(s) with read_file, and "
"this is not one of them. Read it first — a requirement nobody read cannot bind a "
"direction"
)
requirements.append(DeclaredRequirement(bundle_id=bundle_id, path=path, ref=ref))
return {"declared": True, "bundle_id": bundle_id, "path": path, "ref": ref}
tools = [list_bundles, read_bundle, read_dir, read_file]
if requirements is not None:
tools.append(declare_requirement)
return tools
def requirement_payload(declared: Sequence[DeclaredRequirement]) -> list[dict[str, Any]]:
"""The ONE rendering of declared requirements into plain data, in DECLARATION order.
Two artefacts carry it — ``{run_id}-exploration.json`` and ``{run_id}-debate.json`` — and two
copies of "what a declaration looks like" would drift into two answers about one run, which is
the kø-(p) defect landing in exactly the files an operator reads after a paid run.
"""
return [{"bundle_id": d.bundle_id, "path": d.path, "ref": d.ref} for d in declared]
def _refused(
@ -1369,6 +1484,8 @@ def fresh_exploration_workflow(
bundle_dirs: Sequence[str] = (),
middleware: Sequence[Any] | None = None,
quick_validate_sink: list[QuickValidation] | None = None,
tool_call_sink: list[ToolCall] | None = None,
requirement_sink: list[DeclaredRequirement] | None = None,
checkpoint_dir: str | None = None,
) -> Any:
"""Build a FRESH Magentic workflow with fresh agents and fresh clients (mirrors
@ -1392,9 +1509,18 @@ def fresh_exploration_workflow(
token guarantee: agent-level ``ChatMiddleware`` does fire on the manager's own calls (measured
A1), and the manager is the most talkative participant in the loop.
"""
# P19 DEL A: the declaration tool reaches BOTH roles, and that is a measurement rather than
# generosity. The instruction that asks for a binding requirement is the HYPOTHESISER's — it is
# the role that commits to a direction — while the ``read_file`` calls the refusal checks are
# the NAVIGATOR's. Giving it to the navigator alone would leave the committing role unable to
# state its own commitment; to the hypothesiser alone, unable to declare what its partner read.
navigator = list(
navigator_tools(bundle_dirs, opened=tool_call_sink, requirements=requirement_sink)
)
hypothesiser_tools: list[Any] = [quick_validate_tool(bundle_dirs, sink=quick_validate_sink)]
hypothesiser_tools += [t for t in navigator if getattr(t, "name", "") == "declare_requirement"]
tools_by_role: dict[str, list[Any]] = {
NAVIGATOR_ROLE: list(navigator_tools(bundle_dirs)),
NAVIGATOR_ROLE: navigator,
HYPOTHESISER_ROLE: hypothesiser_tools,
}
participants = [
@ -1583,13 +1709,45 @@ def _resolve_hypothesis_bundle(raw: str, bundle_ids: Sequence[str]) -> str:
)
def _requirement_of(data: Mapping[str, Any], line: str) -> BindingRequirement | None:
"""The marked hypothesis's binding requirement, or ``None`` when the base genuinely has none.
**The field is never OMITTED** (P19 A1): a missing key is a hard error of the same class as an
unreadable marked line, because the marker is what makes fail-closed affordable — the model
committed to a direction, and "which requirement binds it" is part of that commitment rather
than an optional extra. An EXPLICIT ``null`` is legal and needs ``why_none``: a base that holds
no requirement for a direction is a finding worth stating, and one stated without a reason is
indistinguishable from the model having skipped the question.
"""
if "requirement" not in data:
raise HypothesisParseError(
"a marked hypothesis must carry a 'requirement' — either "
'{"path": ..., "ref": ...} for the document that binds it, or null together with '
f"'why_none'. A direction with no requirement behind it is a guess; got: {line}"
)
raw = data["requirement"]
if raw is None:
if not data.get("why_none"):
raise HypothesisParseError(
"a marked hypothesis with 'requirement': null must say 'why_none' — the base "
f"holding no requirement is a finding, and an unexplained null is a silence: {line}"
)
return None
if not isinstance(raw, dict) or not raw.get("path") or not raw.get("ref"):
raise HypothesisParseError(
"a marked hypothesis's 'requirement' needs a non-empty 'path' and 'ref'; a half-named "
f"requirement reads as a citation and points at nothing: {line}"
)
return BindingRequirement(path=str(raw["path"]), ref=str(raw["ref"]))
def _parse_hypotheses(
texts: Sequence[str], bundle_ids: Sequence[str]
) -> list[tuple[str, str, str]]:
"""Every marked ``(label, rationale, bundle_id)`` triple the hypothesiser committed to, in turn
order. The base is resolved HERE rather than at minting time so an unroutable claim is refused
while the line that made it is still in hand for the error message."""
found: list[tuple[str, str, str]] = []
) -> list[tuple[str, str, str, BindingRequirement | None]]:
"""Every marked ``(label, rationale, bundle_id, requirement)`` the hypothesiser committed to,
in turn order. The base is resolved HERE rather than at minting time so an unroutable claim is
refused while the line that made it is still in hand for the error message."""
found: list[tuple[str, str, str, BindingRequirement | None]] = []
for text in texts:
for line in text.splitlines():
stripped = line.strip()
@ -1613,6 +1771,7 @@ def _parse_hypotheses(
str(data["label"]),
str(data["rationale"]),
_resolve_hypothesis_bundle(str(data.get("bundle_id") or ""), bundle_ids),
_requirement_of(data, stripped),
)
)
return found
@ -1649,7 +1808,8 @@ def refuse_unroutable_seeds(seeds: Sequence[Approach], bundle_ids: Sequence[str]
def _mint_approaches(
seeds: Sequence[Approach], discovered: Sequence[tuple[str, str, str]]
seeds: Sequence[Approach],
discovered: Sequence[tuple[str, str, str, BindingRequirement | None]],
) -> tuple[Approach, ...]:
"""Seeds FIRST, untouched, then one approach per discovered direction.
@ -1662,7 +1822,7 @@ def _mint_approaches(
taken = {approach.id for approach in seeds}
minted: list[Approach] = list(seeds)
counter = 0
for label, rationale, bundle_id in discovered:
for label, rationale, bundle_id, requirement in discovered:
counter += 1
candidate = f"hypothesis-{counter}"
while candidate in taken or candidate == OWN_PROPOSAL_ID:
@ -1675,8 +1835,17 @@ def _mint_approaches(
#
# ``bundle_id`` is stamped on a MINTED approach because there is nothing here to preserve —
# the opposite call from the seeds above, which pass through untouched (§ C.6 door 1).
# ``requirement`` is stamped on a MINTED approach for the same reason ``bundle_id`` is:
# there is nothing here to preserve. A SEED passes through untouched (§ C.6 door 1) — an
# expert who named no requirement is not to be given one on their behalf.
minted.append(
Approach(id=candidate, label=label, description=rationale, bundle_id=bundle_id)
Approach(
id=candidate,
label=label,
description=rationale,
bundle_id=bundle_id,
requirement=requirement,
)
)
return tuple(minted)
@ -1801,6 +1970,11 @@ async def explore(
bundle_dirs=bundle_dirs,
middleware=[BudgetMiddleware(meter), ExplorationToolRecorder(trace.tool_calls)],
quick_validate_sink=trace.quick_validations,
# ALIASES of the trace's own lists, never copies (the ``_drive`` rule): the refusal reads
# the same read trace the recorder writes, so the two can never disagree about what this
# run opened.
tool_call_sink=trace.tool_calls,
requirement_sink=trace.requirements,
checkpoint_dir=checkpoint_dir,
)
@ -2093,6 +2267,11 @@ async def resume_exploration(
bundle_dirs=parked.bundle_dirs,
middleware=[BudgetMiddleware(meter), ExplorationToolRecorder(trace.tool_calls)],
quick_validate_sink=trace.quick_validations,
# ALIASES of the trace's own lists, never copies (the ``_drive`` rule): the refusal reads
# the same read trace the recorder writes, so the two can never disagree about what this
# run opened.
tool_call_sink=trace.tool_calls,
requirement_sink=trace.requirements,
checkpoint_dir=checkpoint_dir,
)