feat(s2c): debatten navigerer basen i stedet for aa faa den utlevert [skip-docs]
MAJOR-3/S7a-3 gjorde utforskningen billig og lot pipelinen staa. Maalt paa K2
(630 konsepter, S7bs eget instrument, kjent-positiv-kontrollen reprodusert
eksakt FOER bruk): okf.bundle_context er 648 962 o200k-tokens og rir i TRE
kopier = 1 947 342 = 99,1 % av en kjoerings prompt-tokens.
Et premiss i maaledokumentet ble presisert foerst: de tre kopiene er tre
DEBATT-turer (proposer x2, checker x1), mens genererings-prompten er 156
tokens, fordi gen_context = debate_output or context. Det avgjorde formen -
generering trengte ingen egen soem, for aa binde `context` binder
siste-utvei-fallbacken ved konstruksjon.
run_project sender naa en PEKER (fast tekst + erklaert bundle_id + antall
konseptdokumenter i scope + stigen, O(1) i korpuset) og gir debatten de SAMME
fire verktoeyene utforskningen bruker - explore.navigator_tools gjenbrukt,
aldri en andre kopi av policyen.
Etter: 753 tokens like-for-like (samme manus, samme fire prompter, -99,96 %)
og 8 942 med en debatt som faktisk gaar stigen (-99,5 %), mot operatoerens
terskel 195 000 = 4,6 % av taket. Validert besparelse og validatorens dom er
UENDRET (850 000 NOK av 3 852 500, 2 av 5 felt paa stage 4 og 5, samme
dom-noekkel), og utforskningens 18 355 er uendret til tokenet.
§4.1a maatte flytte, ikke forsvinne: dimensjonsfilteret bodde i renderingen og
bor naa i VERKTOEYENE, paa begge trinn - en listing som skjuler et fremmed
dokument mens read_file serverer det paa sti er et filter i navnet alene.
okf.in_dimension er eneste predikat.
Sporet er kaller-eid (ExplorationToolRecorder -> RunResult.debate_tool_calls ->
{run_id}-debate.json fra en finally) og skrives ogsaa TOMT: en debatt som
navigerer ingenting ER S2c-regresjonen, saa den maa kunne leses.
Load-bearing MAALT: aatte mutasjoner roede mot HELE suiten, groenn kontroll
1306/5 (fra 1295/5), golden demo-transcript.stdout BYTE-UENDRET
(shasum -a 1 av innholdet = ea8c534773acdbe41ae68f2c55724d69aaf8be4f).
M7 falsifiserte seg selv, ikke gaten - staar som maalt.
Maaling: docs/2026-09-04-s2c-debatt-k2.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
5d8844fef5
commit
da5f10f140
13 changed files with 1044 additions and 98 deletions
|
|
@ -60,18 +60,22 @@ from portfolio_optimiser.explore import (
|
|||
NAVIGATOR_ROLE,
|
||||
ExplorationContract,
|
||||
ExplorationResult,
|
||||
ExplorationToolRecorder,
|
||||
ExplorationTrace,
|
||||
ParkedStateError,
|
||||
PlanReviewDecision,
|
||||
PlanReviewParked,
|
||||
ToolCall,
|
||||
explore,
|
||||
exploration_notice,
|
||||
load_exploration_contract,
|
||||
load_parked,
|
||||
parked_notice,
|
||||
parked_payload,
|
||||
navigator_tools,
|
||||
resume_exploration,
|
||||
terminal_plan_reviewer,
|
||||
tool_call_payload,
|
||||
trace_payload,
|
||||
)
|
||||
from portfolio_optimiser.generate import ParseFailure, generate_via_llm
|
||||
|
|
@ -199,6 +203,16 @@ class RunResult:
|
|||
#: ever report zero — unlike ``cost_baseline_anchored`` and ``skipped_links``, both resolved
|
||||
#: above that cut.
|
||||
unkeyed_verdicts: int = 0
|
||||
#: Which documents the DEBATE opened, in call order (S2c). Since the debate navigates the base
|
||||
#: instead of being handed ``bundle_context``, "what did this run read" is no longer answerable
|
||||
#: from the prompts — that is the whole saving — so the run carries the trace itself.
|
||||
#:
|
||||
#: EMPTY is an honest POSITIVE statement ("the debate opened nothing"), which is why it
|
||||
#: defaults, exactly as ``skipped_links`` does; and it is also the S2c regression signal, which
|
||||
#: is why the outbox artefact is written even when it is empty rather than only on activity.
|
||||
#: The RESULT of each call is deliberately absent — that is the base's content, i.e. the very
|
||||
#: thing too big to ride along (``ToolCall``'s own rule, MAJOR-1).
|
||||
debate_tool_calls: tuple[ToolCall, ...] = ()
|
||||
|
||||
@property
|
||||
def verdict_key(self) -> str:
|
||||
|
|
@ -500,6 +514,38 @@ def _authored_texts(result: Any, name: str) -> list[str]:
|
|||
return texts
|
||||
|
||||
|
||||
#: The pointer's shape is fixed text plus the base id, its document count and (when scoped) the
|
||||
#: dimension — O(1) in the corpus by construction. The ceiling that guards it lives in the TEST
|
||||
#: (``test_debate_navigation_cost_loadbearing``), for ``_CATALOGUE_EXCERPT_CHARS``' reason: a bound
|
||||
#: imported from the implementation moves with it, and widening this is the regression the gate
|
||||
#: exists to catch.
|
||||
def _bundle_pointer(bundle: okf.Bundle, bundle_id: str, *, dimension: str | None = None) -> str:
|
||||
"""What the debate is told about the knowledge base INSTEAD of being given it (S2c).
|
||||
|
||||
It must do exactly two things: NAME the base by the id the tools take — a bounded prompt that
|
||||
omits it is a debate that cannot make a single call, which is this seam's vacuous form — and
|
||||
say the ladder exists. It carries no content: the whole point is that the corpus is read on
|
||||
demand and each result rides only from the call that asked for it.
|
||||
|
||||
The document COUNT is the price signal a proposer chooses against (``directory_listing``'s own
|
||||
``chars`` rule, one rung up), and it is the count IN SCOPE: under a dimension, advertising
|
||||
documents the tools will then refuse would be a number that describes a different run.
|
||||
"""
|
||||
documents = sum(1 for f in bundle.context_files if okf.in_dimension(f, dimension))
|
||||
scope = (
|
||||
f" Scope: dimension {dimension!r} — only knowledge in that scope is readable."
|
||||
if dimension
|
||||
else ""
|
||||
)
|
||||
return (
|
||||
f"Knowledge base: {bundle_id} ({documents} concept documents)."
|
||||
f"{scope}\n"
|
||||
"It is NOT included here — read it with your tools: list_bundles() for the bases, "
|
||||
f"read_bundle({bundle_id!r}) for its top level, read_dir({bundle_id!r}, path) for one "
|
||||
f"directory, read_file({bundle_id!r}, path) for one document. Open what you need."
|
||||
)
|
||||
|
||||
|
||||
def _debate_text(result: Any) -> str:
|
||||
"""The PROPOSER's converged output (fed into generation, F1). With ``output_from=agents`` both
|
||||
participants surface, so we select proposer-authored outputs specifically — taking the last of
|
||||
|
|
@ -844,8 +890,10 @@ async def run_project(
|
|||
# 2-3. Project + agent read-context + first-class citations. A bundle run derives ALL THREE from
|
||||
# the navigated OKF bundle via progressive disclosure (verdict layer EXCLUDED — målbilde §2/§4),
|
||||
# NOT keyword chunk-stuffing; the road path keeps the chunk-retrieval data source. ``debate_tools``
|
||||
# is the query-time retrieval surface — empty on the bundle path (navigation already placed the
|
||||
# curated context in the prompt, and a docs_dir==bundle_dir tool would re-leak the verdict layer).
|
||||
# is the query-time retrieval surface: since S2c the bundle path carries the four NAVIGATOR
|
||||
# tools there (progressive disclosure taken to its conclusion — the agents open what they need
|
||||
# instead of being handed the base), and a ``docs_dir==bundle_dir`` chunk tool is still refused,
|
||||
# because that one would re-leak the verdict layer the navigation excludes by construction.
|
||||
# S4.0 (F3): the run path SETS the validator's cost baseline, so the deterministic gate is
|
||||
# anchored to the project's real cost lines instead of the ones the proposal asserts.
|
||||
# * road path: the reference project's own ``cost_items`` ARE the baseline -> always anchored.
|
||||
|
|
@ -858,7 +906,8 @@ async def run_project(
|
|||
# the mount is carried alongside, so a base delivered under a directory name of its own is
|
||||
# opened rather than refused. What is still refused, before a single model call: a base
|
||||
# whose concepts declare two different corpora.
|
||||
resolved_bundle_id: okf.ResolvedBundleId | None = okf.reconcile_bundle_id(bundle_dir)
|
||||
resolved = okf.reconcile_bundle_id(bundle_dir)
|
||||
resolved_bundle_id: okf.ResolvedBundleId | None = resolved
|
||||
okf.assert_declared_ids_agree(bundle)
|
||||
project = _project_from_bundle(bundle_dir, project_id, bundle=bundle)
|
||||
# The THIRD projection into ``CostBaseline`` (MAJOR-4), behind an EXPLICIT commission and
|
||||
|
|
@ -872,14 +921,22 @@ async def run_project(
|
|||
if derive_cost_baseline
|
||||
else okf.load_optional_cost_baseline(bundle_dir)
|
||||
)
|
||||
# §4.1a context-scope: agents read ONLY dimension-scoped bundle knowledge (Step-3 filter);
|
||||
# dimension=None keeps the full context, byte-identical to before.
|
||||
context = okf.bundle_context(bundle, dimension=dimension.id if dimension else None)
|
||||
# S2c: the debate NAVIGATES the base; it is never handed the whole of it. Measured on K2
|
||||
# (630 concepts, docs/2026-09-04-syretest-s7b-k2.md § 3.4) the rendered context was 648 962
|
||||
# o200k tokens riding in THREE prompts — 99,1 % of a run's prompt cost, none of it asked
|
||||
# for twice. The task message now carries a POINTER, and the agents get the SAME four
|
||||
# tools the exploration uses; ``gen_context = debate_output or context`` below means the
|
||||
# generation fallback is bounded by the same change rather than by a second policy.
|
||||
dimension_id = dimension.id if dimension else None
|
||||
context = _bundle_pointer(bundle, resolved.id, dimension=dimension_id)
|
||||
citations = bundle_citations(bundle)
|
||||
# What the navigation could NOT reach, taken from the run's ONE walk. The road path below
|
||||
# navigates no bundle at all, so its empty tuple is literally true rather than a stand-in.
|
||||
skipped_links: tuple[okf.SkippedLink, ...] = bundle.skipped
|
||||
debate_tools: list[Any] = []
|
||||
# §4.1a context-scope, carried over: the agents read ONLY dimension-matched knowledge. The
|
||||
# filter used to live in the rendering; with navigation it lives in the TOOLS, on both
|
||||
# rungs (``navigator_tools``' own gate), because that is now where the bytes leave.
|
||||
debate_tools: list[Any] = list(navigator_tools([bundle_dir], dimension=dimension_id))
|
||||
else:
|
||||
project = _project_by_id(project_id)
|
||||
baseline = baseline_from_project(project)
|
||||
|
|
@ -915,12 +972,23 @@ async def run_project(
|
|||
# called. Attached only when servers are configured — with none there is nothing to attribute a
|
||||
# call to, and the middleware list stays exactly what it was before Trekk B.
|
||||
call_recorder = ToolCallRecorder(tool_server_index(mcp_servers)) if mcp_servers else None
|
||||
# S2c: a CALLER-OWNED sink for what the debate opens (the ``parse_failures``/``ExplorationTrace``
|
||||
# shape). A returned value would be lost on exactly the run that most needs the evidence — a
|
||||
# budget stop mid-debate raises out of ``debate.run`` and constructs no ``RunResult`` at all.
|
||||
# ``ExplorationToolRecorder`` is REUSED rather than re-implemented: it is already the recorder
|
||||
# for in-process navigator calls, ordered and un-deduplicated, which is exactly the question
|
||||
# here too ("did this run open anything, and in what sequence"). Its sibling
|
||||
# ``mcp_tools.ToolCallRecorder`` stays what it is — a sorted, de-duplicated EGRESS claim.
|
||||
debate_tool_calls: list[ToolCall] = []
|
||||
debate_middleware: list[Any] = [budget_mw, ExplorationToolRecorder(debate_tool_calls)]
|
||||
if call_recorder is not None:
|
||||
debate_middleware.append(call_recorder)
|
||||
debate = fresh_workflow(
|
||||
factory,
|
||||
max_rounds=max_rounds,
|
||||
enable_layer1_hitl=enable_layer1_hitl,
|
||||
tools=debate_tools,
|
||||
middleware=[budget_mw] if call_recorder is None else [budget_mw, call_recorder],
|
||||
middleware=debate_middleware,
|
||||
)
|
||||
# S4.2 cut (comparison protocol §4 pkt 2/3): everything above is offline — contracts, budget, and
|
||||
# the EAGER client build (fresh_workflow constructs the proposer+checker clients, workflow.py:64).
|
||||
|
|
@ -957,12 +1025,23 @@ async def run_project(
|
|||
# opens a connection — its promise to stop before the first call covers egress too. Constructed
|
||||
# tools that are never entered expose nothing, and ones never exited leave the process hanging,
|
||||
# so the stack owns both halves.
|
||||
async with AsyncExitStack() as mcp_stack:
|
||||
for live_tool in live_mcp_tools:
|
||||
await mcp_stack.enter_async_context(live_tool)
|
||||
result = await debate.run(
|
||||
f"Find a cost-saving measure for {project.id}.\nContext:\n{context}"
|
||||
)
|
||||
try:
|
||||
async with AsyncExitStack() as mcp_stack:
|
||||
for live_tool in live_mcp_tools:
|
||||
await mcp_stack.enter_async_context(live_tool)
|
||||
result = await debate.run(
|
||||
f"Find a cost-saving measure for {project.id}.\nContext:\n{context}"
|
||||
)
|
||||
finally:
|
||||
# ``finally``, the ``write_parse_failures`` precedent: any exception leaving the debate —
|
||||
# a budget stop is today's known one — destroys the same evidence, and a list of exception
|
||||
# types is a list that goes stale. Written even when EMPTY: "the debate opened nothing" is
|
||||
# the S2c regression itself, so it must be readable rather than inferred from an absence.
|
||||
if outbox_dir is not None:
|
||||
assert run_id is not None # narrowed by the step-0 guard (no wall-clock default)
|
||||
outbox.write_debate_tools(
|
||||
outbox_dir, run_id, tool_calls=tool_call_payload(debate_tool_calls)
|
||||
)
|
||||
# F1: the candidate must derive from the DEBATE. Feed the proposer's converged output into
|
||||
# generation (retrieval context is the last-resort fallback only). The checker's verdict
|
||||
# (Step 3/4) is parsed from the SAME debate result and gates the outcome below.
|
||||
|
|
@ -1219,6 +1298,7 @@ async def run_project(
|
|||
refinements=tuple(refinements),
|
||||
skipped_links=skipped_links,
|
||||
unkeyed_verdicts=unkeyed_verdicts,
|
||||
debate_tool_calls=tuple(debate_tool_calls),
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -1940,12 +2020,25 @@ def _load_scripted_replies(
|
|||
``scripted_factory``'s lookup, mid-run, long after the run appeared to start cleanly (MAJOR-2:
|
||||
measured for the three ``explore()`` adds on top of the debate's own two).
|
||||
|
||||
**An EXPLORATION role may also be given a step LIST** (MAJOR-1 b), because a single constant
|
||||
string can never emit a ``function_call`` — measured: 0 tool calls / 0 approaches / 1 round on
|
||||
4/4 bases, an offline rehearsal that was vacuous by construction. The list form is refused for
|
||||
the debate's roles BY NAME rather than accepted and ignored: the proposer answers
|
||||
``generate``'s own call, not an agent loop that would invoke a tool between turns, so a script
|
||||
of calls there describes a rehearsal that cannot happen."""
|
||||
**ANY role may be given a step LIST** (MAJOR-1 b), because a single constant string can never
|
||||
emit a ``function_call`` — measured: 0 tool calls / 0 approaches / 1 round on 4/4 bases, an
|
||||
offline rehearsal that was vacuous by construction.
|
||||
|
||||
Until S2c the list form was REFUSED for the debate's two roles, on the stated ground that "the
|
||||
proposer answers ``generate``'s own call, not an agent loop that would invoke a tool between
|
||||
turns, so a script of calls there describes a rehearsal that cannot happen". That ground is now
|
||||
measurably false: the debate's proposer and checker are ``Agent``s in a GroupChat and they hold
|
||||
the four navigator tools, so a constant-string rehearsal proves the debate RUNS while proving
|
||||
nothing about whether it OPENS the base — which is the identical vacuity MAJOR-1 closed one
|
||||
surface over. Keeping the refusal would have made the free half of the measurement ladder
|
||||
unable to reach the very seam S2c builds.
|
||||
|
||||
**Honesty limit, stated rather than encoded.** Scripts are per-CLIENT and each client gets its
|
||||
own copy, so a ``proposer`` script is consumed once by the DEBATE client and again, from the
|
||||
start, by the fresh client ``generate_via_llm`` builds. A script whose first step is a tool
|
||||
call therefore answers the generation call with a ``function_call`` too, which will not parse.
|
||||
That is the operator's to write correctly: guessing which steps were "meant for" which call
|
||||
site would be repair, and this loader validates."""
|
||||
try:
|
||||
raw = json.loads(Path(path).read_text(encoding="utf-8"))
|
||||
except FileNotFoundError as exc:
|
||||
|
|
@ -1963,12 +2056,6 @@ def _load_scripted_replies(
|
|||
for role in required_roles:
|
||||
if isinstance(raw[role], str):
|
||||
continue
|
||||
if role not in _EXPLORATION_SCRIPTED_ROLES:
|
||||
raise ValueError(
|
||||
f"--scripted-replies[{role!r}] is a step list, but that form is the exploration's: "
|
||||
f"only {', '.join(_EXPLORATION_SCRIPTED_ROLES)} run inside an agent loop that can "
|
||||
f"invoke a tool between turns ({path})"
|
||||
)
|
||||
_validate_script(role, raw[role], path)
|
||||
return {role: raw[role] for role in required_roles}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue