feat(fase3): make the maker-checker checker actually gate the reasoning
Closes gap #3 (maalbilde §5): the GroupChat checker critiqued into the void — output_from=[proposer] surfaced only the proposer, so an explicit checker rejection was ignored and the deterministic validator was the sole gate. Two falsifiers now act on the same candidate: the validator gates the NUMBERS (blocking, unchanged), the checker gates the REASONING (maalbilde §2/§6). - workflow.py: output_from=agents surfaces both participants; the checker instruction ends with a VERDICT: APPROVE / VERDICT: REJECT - <reason> line. - run.py: _authored_texts() reads author_name through out.messages (MAF 1.9.0 puts it there, not on the AgentResponse); _debate_text() now selects the PROPOSER-authored output (fixes a latent texts[-1] regression that would feed the checker's verdict to generation at even round counts); _checker_verdict() parses the gate decision. An explicit REJECT overrides an otherwise-validated outcome to a checker-sourced Rejection. Opt-in-reject (fail-open on a missing marker). RunResult gains checker_verdict; provenance.validator_decision is stamped from the validator outcome BEFORE the override, so it never conflates the two falsifiers (provenance honesty). Load-bearing (maalbilde §7): tests/test_checker_gate_loadbearing.py is a PAIR — an explicit checker REJECT on a VALIDATOR-VALID proposal yields a Rejection whose reason carries the checker's reason while validator_decision stays "validated"; the causality control (checker APPROVE, same proposer) validates normally. Proven RED on BOTH detach points (revert output_from, or drop the override). Suite 134->136 passed, 4 skipped; mypy + ruff check clean. Pre-existing ruff-format drift (backends/budget/verdicts/test_contracts) left untouched for a surgical diff. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MHR8iKxJRxDiDfNw8HZmWE
This commit is contained in:
parent
8814a698c2
commit
4ec778c855
5 changed files with 194 additions and 32 deletions
|
|
@ -60,8 +60,11 @@ from portfolio_optimiser.workflow import fresh_workflow
|
|||
@dataclass(frozen=True)
|
||||
class RunResult:
|
||||
"""The outcome of one project run: the validated/rejected proposal, its first-class
|
||||
provenance, the captured (Layer-2) verdict, the ExpeL hits surfaced for it, the store, and
|
||||
the debate's converged output that the candidate was generated from (F1 traceability)."""
|
||||
provenance, the captured (Layer-2) verdict, the ExpeL hits surfaced for it, the store, the
|
||||
debate's converged output that the candidate was generated from (F1 traceability), and the
|
||||
checker's gate decision (Step 3/4: ``"approve" | "reject" | "absent"``). ``checker_verdict``
|
||||
records the checker's decision distinctly from ``provenance.validator_decision`` so the two
|
||||
falsifiers (reasoning vs numbers) are never conflated."""
|
||||
|
||||
outcome: ValidatedProposal | Rejection
|
||||
provenance: ProvenanceStamp
|
||||
|
|
@ -69,6 +72,7 @@ class RunResult:
|
|||
retrieved: list[Verdict]
|
||||
store: VerdictStore
|
||||
debate_output: str
|
||||
checker_verdict: str = "absent"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
|
@ -90,24 +94,47 @@ class PortfolioResult:
|
|||
sum_token_usage: int
|
||||
|
||||
|
||||
# The GroupChat orchestrator emits this terminal notice as a participant output; it is NOT the
|
||||
# debate's substantive result, so the F1 extractor filters it out (verified against MAF 1.9.0).
|
||||
_ORCH_NOTICE = "reached the maximum number of rounds"
|
||||
def _authored_texts(result: Any, name: str) -> list[str]:
|
||||
"""The texts of ``get_outputs()`` entries authored by participant ``name`` (proposer/checker),
|
||||
in surfaced order. MAF surfaces ``author_name`` on each output's ``messages`` — NOT on the
|
||||
``AgentResponse`` itself (verified against 1.9.0) — so we match through ``messages``. This
|
||||
separates the proposer's converged output (fed to generation, F1) from the checker's gate
|
||||
verdict (Step 3/4); the orchestrator's termination notice is authored by neither, so it is
|
||||
excluded automatically."""
|
||||
texts: list[str] = []
|
||||
for out in result.get_outputs():
|
||||
if not any(getattr(m, "author_name", None) == name for m in getattr(out, "messages", [])):
|
||||
continue
|
||||
text = out if isinstance(out, str) else getattr(out, "text", None)
|
||||
if text:
|
||||
texts.append(text)
|
||||
return texts
|
||||
|
||||
|
||||
def _debate_text(result: Any) -> str:
|
||||
"""Extract the proposer's converged output from a ``WorkflowRunResult`` (surfaced by
|
||||
``output_from=[proposer]``), excluding the orchestrator's termination notice. Returns ``""``
|
||||
when no substantive participant text is present."""
|
||||
texts: list[str] = []
|
||||
for out in result.get_outputs():
|
||||
text = out if isinstance(out, str) else getattr(out, "text", None)
|
||||
if not text:
|
||||
continue
|
||||
if getattr(out, "author_name", None) == "group_chat_orchestrator" or _ORCH_NOTICE in text:
|
||||
continue
|
||||
texts.append(text)
|
||||
return texts[-1] if texts else ""
|
||||
"""The PROPOSER's converged output (fed into generation, F1). With ``output_from=agents`` both
|
||||
participants surface, so we select proposer-authored outputs specifically — taking the last of
|
||||
ALL surfaced outputs would feed the checker's verdict to generation at even round counts.
|
||||
Returns ``""`` when the proposer produced no surfaced text."""
|
||||
proposer_texts = _authored_texts(result, "proposer")
|
||||
return proposer_texts[-1] if proposer_texts else ""
|
||||
|
||||
|
||||
def _checker_verdict(result: Any) -> tuple[str, str]:
|
||||
"""Parse the checker's gate verdict from its surfaced debate output (Step 3/4, målbilde §2/§6).
|
||||
Returns ``(decision, reason)``: ``"reject"`` ONLY on an explicit ``VERDICT: REJECT`` (with the
|
||||
trailing reason), ``"approve"`` on an explicit ``VERDICT: APPROVE``, else ``"absent"``. The gate
|
||||
is opt-in-reject (fail-open): a missing/unparseable marker never blocks, so the deterministic
|
||||
validator stays the sole gate on those runs."""
|
||||
checker_texts = _authored_texts(result, "checker")
|
||||
text = checker_texts[-1] if checker_texts else ""
|
||||
upper = text.upper()
|
||||
if "VERDICT: REJECT" in upper:
|
||||
reason = text[upper.index("VERDICT: REJECT") + len("VERDICT: REJECT") :]
|
||||
return "reject", reason.lstrip(" -:—").strip()
|
||||
if "VERDICT: APPROVE" in upper:
|
||||
return "approve", ""
|
||||
return "absent", ""
|
||||
|
||||
|
||||
def _project_by_id(project_id: str) -> Project:
|
||||
|
|
@ -232,8 +259,10 @@ async def run_project(
|
|||
)
|
||||
result = await debate.run(f"Find a cost-saving measure for {project.id}.\nContext:\n{context}")
|
||||
# F1: the candidate must derive from the DEBATE. Feed the proposer's converged output into
|
||||
# generation (retrieval context is the last-resort fallback only).
|
||||
# generation (retrieval context is the last-resort fallback only). The checker's verdict
|
||||
# (Step 3/4) is parsed from the SAME debate result and gates the outcome below.
|
||||
debate_output = _debate_text(result)
|
||||
checker_decision, checker_reason = _checker_verdict(result)
|
||||
gen_context = debate_output or context
|
||||
|
||||
# Step-1 ExpeL wiring (Fase 2a, målbilde §5/§7): fold the candidate's prior verdicts INTO the
|
||||
|
|
@ -247,15 +276,17 @@ async def run_project(
|
|||
fewshot = ExpeLContextProvider(store, expel_query, k=top_k).format_fewshot()
|
||||
gen_context = f"{fewshot}\n\n{gen_context}"
|
||||
|
||||
# 5. Structured candidate -> blocking validation; token bound = the meter in this loop.
|
||||
# 5. Structured candidate -> blocking validation on the NUMBERS; token bound = the meter.
|
||||
proposer_client = factory("proposer")
|
||||
outcome = await generate_via_llm(proposer_client, project, gen_context, meter)
|
||||
proposal = outcome.proposal
|
||||
validator_outcome = await generate_via_llm(proposer_client, project, gen_context, meter)
|
||||
proposal = validator_outcome.proposal
|
||||
|
||||
# 6. First-class provenance stamp (authoritative; independent of MAF Annotation).
|
||||
# F1: an injected client_factory stamps the injected client's REAL model ("unknown" is the
|
||||
# neutral fallback for a client that doesn't surface one -- never a fabricated name); the
|
||||
# default path keeps the deterministic resolve_model.
|
||||
# default path keeps the deterministic resolve_model. validator_decision reflects the VALIDATOR
|
||||
# (the numbers) ONLY -- stamped from validator_outcome BEFORE the checker override below, so a
|
||||
# checker-gated proposal whose numbers passed is never mislabelled as validator-rejected.
|
||||
model = (
|
||||
(getattr(proposer_client, "model", None) or "unknown")
|
||||
if client_factory is not None
|
||||
|
|
@ -265,10 +296,21 @@ async def run_project(
|
|||
citations=citations,
|
||||
model=model,
|
||||
role="proposer",
|
||||
validator_decision="validated" if isinstance(outcome, ValidatedProposal) else "rejected",
|
||||
validator_decision=(
|
||||
"validated" if isinstance(validator_outcome, ValidatedProposal) else "rejected"
|
||||
),
|
||||
token_usage=meter.tokens,
|
||||
)
|
||||
|
||||
# 6b. Step 3/4 checker gate (målbilde §2/§6): the validator falsifies the numbers, the checker
|
||||
# falsifies the reasoning. An explicit checker REJECT blocks an otherwise-validated proposal; a
|
||||
# validator rejection (the numbers) already stands. Fail-open: APPROVE/absent never blocks.
|
||||
outcome: ValidatedProposal | Rejection
|
||||
if isinstance(validator_outcome, ValidatedProposal) and checker_decision == "reject":
|
||||
outcome = Rejection(proposal=proposal, reason=f"checker rejected: {checker_reason}")
|
||||
else:
|
||||
outcome = validator_outcome
|
||||
|
||||
# 7. ExpeL (regression guard + traceability): exercises the two-arg extend_instructions
|
||||
# injection on a REAL SessionContext (the Critical Fase-1 GA-signature guard), and surfaces
|
||||
# the proposal-keyed retrieval for RunResult.retrieved. On the bundle path the load-bearing
|
||||
|
|
@ -294,6 +336,7 @@ async def run_project(
|
|||
retrieved=retrieved,
|
||||
store=store,
|
||||
debate_output=debate_output,
|
||||
checker_verdict=checker_decision,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -26,7 +26,12 @@ from agent_framework.orchestrations import GroupChatBuilder
|
|||
_MAKER_CHECKER_ROLES = ("proposer", "checker")
|
||||
_INSTRUCTIONS = {
|
||||
"proposer": "You propose one concrete cost-saving measure for the project.",
|
||||
"checker": "You critique the proposal and flag any constraint violation.",
|
||||
"checker": (
|
||||
"You critique the proposal's reasoning and flag any constraint violation. "
|
||||
"End your reply with exactly one verdict line: 'VERDICT: APPROVE' if the reasoning "
|
||||
"holds, or 'VERDICT: REJECT - <short reason>' if it does not. The validator gates the "
|
||||
"numbers; your verdict gates the reasoning (Step 3/4 — målbilde §2)."
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -79,10 +84,11 @@ def fresh_workflow(
|
|||
is called once per role, so each run owns its own clients — no state survives between runs.
|
||||
|
||||
``tools`` + ``middleware`` are attached to each agent (F2/F7; constructed by the
|
||||
orchestrator). ``output_from=[proposer]`` makes ``WorkflowRunResult.get_outputs()`` surface
|
||||
the proposer's converged output — without it, ``get_outputs()`` yields only the
|
||||
orchestrator's "reached max rounds" notice, so the F1 debate->generation dataflow could not
|
||||
read the debate result (verified against installed 1.9.0).
|
||||
orchestrator). ``output_from=agents`` makes ``WorkflowRunResult.get_outputs()`` surface BOTH
|
||||
participants' converged outputs — the proposer's (fed to generation, F1) AND the checker's
|
||||
gate verdict (Step 3/4); ``run.py`` separates them by per-message ``author_name``. Without it,
|
||||
``get_outputs()`` yields only the orchestrator's "reached max rounds" notice (verified against
|
||||
installed 1.9.0).
|
||||
"""
|
||||
agents = maker_checker_agents(client_factory, tools=tools, middleware=middleware)
|
||||
# Agents are built from _MAKER_CHECKER_ROLES in order with name=role, so the role tuple
|
||||
|
|
@ -100,9 +106,10 @@ def fresh_workflow(
|
|||
selection_func=select,
|
||||
# Safety net well above the hard cap; with_max_rounds is the binding bound (B4).
|
||||
termination_condition=make_termination(max_rounds * len(names) + 1),
|
||||
# Surface the PROPOSER's converged output so get_outputs() carries the debate
|
||||
# result (F1); the default surfaces only the orchestrator's termination notice.
|
||||
output_from=[agents[0]],
|
||||
# Surface BOTH participants so get_outputs() carries the proposer's converged output
|
||||
# (fed to generation, F1) AND the checker's gate verdict (Step 3/4); run.py separates
|
||||
# them by author_name. The default surfaces only the orchestrator's termination notice.
|
||||
output_from=agents,
|
||||
).with_max_rounds(max_rounds)
|
||||
if enable_layer1_hitl:
|
||||
# Layer-1: in-run synchronous review on the checker (no checkpoint — research 01).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue