feat(p22): declare_requirement answers with a COMPARISON, not a confirmation
P19 DEL A made a direction name the requirement that binds it; P20/A1 made the reply carry the DOCUMENT's own title and number instead of echoing the caller's arguments. Re-measured at the head of this session against the six round-5 debate traces: requirement_hit is 0 of 20 approach rows and 0 of 12 declarations -- the third round in a row at zero. P21/C1 made the runs LOOK first and it worked on its own terms (distinct documents before a declaration went 1,1,1,2,5,13 -> 3,3,5,7,11,12) and the hit did not move. The runs were made to read MORE, not righter. The reply now compares: it names the directions the run was commissioned to pursue and says which of their words appear in the declared document's own title and number, or that none do. A REPORT, never a gate -- the declaration is recorded either way, because a requirement can bind a measure without sharing a word with the name someone gave it, which is exactly how the alternative rule the C1 measurement rejected failed one rung over. The words compared are the DOCUMENT's, never `ref`: a comparison against the caller's own argument can only ever agree. Matching is generous in both directions, and that failure direction is chosen -- a false "no overlap" pushes a model away from a declaration that was right, a false "overlap" merely keeps the report quiet. MEASURED BEFORE IT WAS BUILT, offline against the six traces as the order required (no paid calls in DEL B): the rule speaks on 10 of 12 declarations and stays quiet on 2. A rule that spoke on 12 of 12, or on 0 of 12, could not tell the two classes apart. `labels` defaults to empty, so every call site written before today is byte-identical and the three keys are ABSENT rather than empty -- "there was nothing to compare against" and "we compared and found nothing" are different facts. RUN-level, as the declaration itself is (P19 A4). Also re-measured: the order cited requirement_hit as "0 of 12". The field is per APPROACH (0 of 20); 12 is the number of DECLARATIONS (7 distinct, 0 hits). Both zero, so the conclusion stands, but they are two populations. Load-bearing MEASURED (tests/test_requirement_comparison_loadbearing.py, 8 arms), eight mutations all red against the WHOLE suite + green control 1881/5 (from 1873/5, superset, 0 removed) and golden demo-transcript.stdout BYTE-UNCHANGED (shasum -a 1 of the CONTENT = ea8c534773acdbe41ae68f2c55724d69aaf8be4f): B1 detach the run.py wiring (1 red, that arm alone) - B2 always report an overlap (5) - B3 never report one (2) - B4 compare against the caller's ref (1, that arm alone) - B5 make it a gate (5) - B6 emit the keys with no directions (2, one an OLDER independent witness) - B7 exact token equality instead of substring (1) - B8 drop the minimum word length (3). Honesty limits, stated: no LIVE model has read the comparison yet (DEL D is the measurement); the report cannot say a requirement IS right, only that it shares no word with the direction; and finding 4 (`named` 1/20) is this same matter from the other side, so DEL D measures it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
0a81de2d76
commit
9072359606
5 changed files with 360 additions and 1 deletions
|
|
@ -24,6 +24,7 @@ This module imports ``agent_framework.orchestrations`` and therefore may never b
|
|||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections.abc import Awaitable, Callable, Mapping, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
|
|
@ -1126,12 +1127,62 @@ def _refused_text(exc: Exception) -> str:
|
|||
return f"REFUSED ({_refusal_kind(exc)}): {exc}"
|
||||
|
||||
|
||||
#: P22 DEL B - the shortest word of a direction's label that can carry meaning into a comparison.
|
||||
#: Below this every label shares "for", "med", "ny" with half a corpus and the report would speak
|
||||
#: of an overlap nobody meant.
|
||||
_LABEL_WORD_MIN: Final = 4
|
||||
|
||||
|
||||
def _label_overlap(labels: Sequence[str], *document_text: str) -> tuple[str, ...]:
|
||||
"""Which words of the commission's directions appear in the DECLARED DOCUMENT's own text.
|
||||
|
||||
The measured defect (P21 funn 2, re-measured at the head of okt 126): ``requirement_hit`` is
|
||||
**0 of 20** approach rows over round 5's six paid runs and **0 of 12** declarations - a third
|
||||
round in a row - and P21/C1 moved the documents read before a declaration from 1,1,1,2,5,13 to
|
||||
3,3,5,7,11,12 without moving the hit. The runs were made to read MORE, not righter. P20/A1
|
||||
already gives back the document's own title and number; what nobody said was whether that
|
||||
document has anything to do with the direction the run is committed to.
|
||||
|
||||
**A REPORT, never a gate.** The declaration is recorded either way. A gate on word overlap
|
||||
would refuse legitimate declarations - a requirement can bind a measure without sharing a word
|
||||
with the name someone gave it - which is the P21/C1 alternative rule's failure, one rung over.
|
||||
|
||||
**Generous in BOTH directions, and that is the failure direction chosen on purpose.** A token
|
||||
matches when it is a substring of a document token or the document token is a substring of it,
|
||||
so ``rundkjoring`` meets ``Rundkjoringer`` and ``asfaltdekke`` meets ``Asfalt``. The report
|
||||
says one of two things, and only one of them can do harm: a false "no overlap" pushes a model
|
||||
away from a declaration that was right, while a false "overlap" merely keeps the report quiet.
|
||||
Substring matching fails towards quiet. (P18's ``filter`` chose the same direction for the same
|
||||
reason: a substring fails towards showing MORE, which can be narrowed.)
|
||||
|
||||
MEASURED offline against the six round-5 debate traces before this was built: the rule speaks
|
||||
on **10 of 12** declarations and stays quiet on 2 (both fv412, on ``materialer``). A rule that
|
||||
spoke on 12 of 12 or on 0 of 12 could not tell the two classes apart, which is the same test
|
||||
P21/C1's threshold had to pass.
|
||||
"""
|
||||
haystack = {
|
||||
token
|
||||
for text in document_text
|
||||
for token in re.split(r"[\W_]+", text.lower())
|
||||
if len(token) >= _LABEL_WORD_MIN
|
||||
}
|
||||
shared: set[str] = set()
|
||||
for label in labels:
|
||||
for token in re.split(r"[\W_]+", label.lower()):
|
||||
if len(token) < _LABEL_WORD_MIN:
|
||||
continue
|
||||
if any(token in other or other in token for other in haystack):
|
||||
shared.add(token)
|
||||
return tuple(sorted(shared))
|
||||
|
||||
|
||||
def navigator_tools(
|
||||
bundle_dirs: Sequence[str],
|
||||
*,
|
||||
dimension: str | None = None,
|
||||
opened: list[ToolCall] | None = None,
|
||||
requirements: list[DeclaredRequirement] | None = None,
|
||||
labels: Sequence[str] = (),
|
||||
) -> list[FunctionTool]:
|
||||
"""The navigator's tools: survey the catalogue, open one base, read one document — and, when
|
||||
the caller offers the two sinks, DECLARE the requirement that binds a direction.
|
||||
|
|
@ -1465,7 +1516,7 @@ def navigator_tools(
|
|||
# ``binds`` says out loud what the declaration is for: without it the reply is data with no
|
||||
# instruction, and the instruction is the whole correction.
|
||||
declared = _declared_document(index, bundle_id, path)
|
||||
return {
|
||||
reply: dict[str, Any] = {
|
||||
"declared": True,
|
||||
"bundle_id": bundle_id,
|
||||
"path": path,
|
||||
|
|
@ -1477,6 +1528,30 @@ def navigator_tools(
|
|||
"declaration of a requirement that is not about the measure is worth nothing."
|
||||
),
|
||||
}
|
||||
# P22 DEL B: turn the reply into a COMPARISON against what this run was commissioned to
|
||||
# do. P20/A1 made the reply carry the document's own words; measured, that was not enough
|
||||
# on its own - three rounds of declarations and not one named a fasit concept. The words
|
||||
# compared are the DOCUMENT's (read off the base), never ``ref``, which is the caller's own
|
||||
# argument echoed back: a comparison against the caller's input can only ever agree.
|
||||
# Absent labels (the exploration mints its own directions, so there are none at
|
||||
# declaration time) the two keys are omitted and the reply is byte-identical to P20's.
|
||||
if labels:
|
||||
overlap = _label_overlap(labels, declared[0], declared[1])
|
||||
listed = ", ".join(repr(label) for label in labels)
|
||||
reply["directions"] = list(labels)
|
||||
reply["overlap"] = list(overlap)
|
||||
reply["compare"] = (
|
||||
f"You declared {declared[0]!r} ({declared[1]!r}) for these directions: {listed}. "
|
||||
+ (
|
||||
f"Words they share: {', '.join(overlap)}."
|
||||
if overlap
|
||||
else "No word of any of them appears in the document's own title or number. "
|
||||
"If this document is not about the measure you declared it for, it is the "
|
||||
"wrong requirement: filter the level again with a word from the direction "
|
||||
"itself and read the candidates that come back."
|
||||
)
|
||||
)
|
||||
return reply
|
||||
|
||||
tools = [list_bundles, read_bundle, read_dir, read_file]
|
||||
if requirements is not None:
|
||||
|
|
|
|||
|
|
@ -1294,6 +1294,12 @@ async def run_project(
|
|||
dimension=dimension_id,
|
||||
opened=debate_tool_calls,
|
||||
requirements=debate_requirements,
|
||||
# P22 DEL B: the commission's own direction names, so the declaration rung can
|
||||
# answer with a COMPARISON instead of a confirmation. RUN-level, exactly as
|
||||
# the declaration is (P19 A4): the debate declares once per run, so the reply
|
||||
# names every direction the run carries rather than picking one it cannot
|
||||
# attribute. Without a mandate this is empty and the reply is unchanged.
|
||||
labels=[a.label for a in mandate.approaches] if mandate else (),
|
||||
)
|
||||
)
|
||||
# What the navigation could NOT reach, taken from the run's ONE walk. The road path below
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue