feat(p22): declare_requirement answers with a COMPARISON, not a confirmation

P19 DEL A made a direction name the requirement that binds it; P20/A1 made the reply carry the
DOCUMENT's own title and number instead of echoing the caller's arguments. Re-measured at the head
of this session against the six round-5 debate traces: requirement_hit is 0 of 20 approach rows and
0 of 12 declarations -- the third round in a row at zero. P21/C1 made the runs LOOK first and it
worked on its own terms (distinct documents before a declaration went 1,1,1,2,5,13 -> 3,3,5,7,11,12)
and the hit did not move. The runs were made to read MORE, not righter.

The reply now compares: it names the directions the run was commissioned to pursue and says which
of their words appear in the declared document's own title and number, or that none do.

A REPORT, never a gate -- the declaration is recorded either way, because a requirement can bind a
measure without sharing a word with the name someone gave it, which is exactly how the alternative
rule the C1 measurement rejected failed one rung over. The words compared are the DOCUMENT's, never
`ref`: a comparison against the caller's own argument can only ever agree. Matching is generous in
both directions, and that failure direction is chosen -- a false "no overlap" pushes a model away
from a declaration that was right, a false "overlap" merely keeps the report quiet.

MEASURED BEFORE IT WAS BUILT, offline against the six traces as the order required (no paid calls
in DEL B): the rule speaks on 10 of 12 declarations and stays quiet on 2. A rule that spoke on 12
of 12, or on 0 of 12, could not tell the two classes apart.

`labels` defaults to empty, so every call site written before today is byte-identical and the three
keys are ABSENT rather than empty -- "there was nothing to compare against" and "we compared and
found nothing" are different facts. RUN-level, as the declaration itself is (P19 A4).

Also re-measured: the order cited requirement_hit as "0 of 12". The field is per APPROACH (0 of 20);
12 is the number of DECLARATIONS (7 distinct, 0 hits). Both zero, so the conclusion stands, but they
are two populations.

Load-bearing MEASURED (tests/test_requirement_comparison_loadbearing.py, 8 arms), eight mutations
all red against the WHOLE suite + green control 1881/5 (from 1873/5, superset, 0 removed) and
golden demo-transcript.stdout BYTE-UNCHANGED (shasum -a 1 of the CONTENT =
ea8c534773acdbe41ae68f2c55724d69aaf8be4f): B1 detach the run.py wiring (1 red, that arm alone) -
B2 always report an overlap (5) - B3 never report one (2) - B4 compare against the caller's ref
(1, that arm alone) - B5 make it a gate (5) - B6 emit the keys with no directions (2, one an OLDER
independent witness) - B7 exact token equality instead of substring (1) - B8 drop the minimum word
length (3).

Honesty limits, stated: no LIVE model has read the comparison yet (DEL D is the measurement); the
report cannot say a requirement IS right, only that it shares no word with the direction; and
finding 4 (`named` 1/20) is this same matter from the other side, so DEL D measures it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-16 00:04:47 +02:00
commit 9072359606
5 changed files with 360 additions and 1 deletions

View file

@ -24,6 +24,7 @@ This module imports ``agent_framework.orchestrations`` and therefore may never b
from __future__ import annotations
import json
import re
import sys
from collections.abc import Awaitable, Callable, Mapping, Sequence
from dataclasses import dataclass, field
@ -1126,12 +1127,62 @@ def _refused_text(exc: Exception) -> str:
return f"REFUSED ({_refusal_kind(exc)}): {exc}"
#: P22 DEL B - the shortest word of a direction's label that can carry meaning into a comparison.
#: Below this every label shares "for", "med", "ny" with half a corpus and the report would speak
#: of an overlap nobody meant.
_LABEL_WORD_MIN: Final = 4
def _label_overlap(labels: Sequence[str], *document_text: str) -> tuple[str, ...]:
"""Which words of the commission's directions appear in the DECLARED DOCUMENT's own text.
The measured defect (P21 funn 2, re-measured at the head of okt 126): ``requirement_hit`` is
**0 of 20** approach rows over round 5's six paid runs and **0 of 12** declarations - a third
round in a row - and P21/C1 moved the documents read before a declaration from 1,1,1,2,5,13 to
3,3,5,7,11,12 without moving the hit. The runs were made to read MORE, not righter. P20/A1
already gives back the document's own title and number; what nobody said was whether that
document has anything to do with the direction the run is committed to.
**A REPORT, never a gate.** The declaration is recorded either way. A gate on word overlap
would refuse legitimate declarations - a requirement can bind a measure without sharing a word
with the name someone gave it - which is the P21/C1 alternative rule's failure, one rung over.
**Generous in BOTH directions, and that is the failure direction chosen on purpose.** A token
matches when it is a substring of a document token or the document token is a substring of it,
so ``rundkjoring`` meets ``Rundkjoringer`` and ``asfaltdekke`` meets ``Asfalt``. The report
says one of two things, and only one of them can do harm: a false "no overlap" pushes a model
away from a declaration that was right, while a false "overlap" merely keeps the report quiet.
Substring matching fails towards quiet. (P18's ``filter`` chose the same direction for the same
reason: a substring fails towards showing MORE, which can be narrowed.)
MEASURED offline against the six round-5 debate traces before this was built: the rule speaks
on **10 of 12** declarations and stays quiet on 2 (both fv412, on ``materialer``). A rule that
spoke on 12 of 12 or on 0 of 12 could not tell the two classes apart, which is the same test
P21/C1's threshold had to pass.
"""
haystack = {
token
for text in document_text
for token in re.split(r"[\W_]+", text.lower())
if len(token) >= _LABEL_WORD_MIN
}
shared: set[str] = set()
for label in labels:
for token in re.split(r"[\W_]+", label.lower()):
if len(token) < _LABEL_WORD_MIN:
continue
if any(token in other or other in token for other in haystack):
shared.add(token)
return tuple(sorted(shared))
def navigator_tools(
bundle_dirs: Sequence[str],
*,
dimension: str | None = None,
opened: list[ToolCall] | None = None,
requirements: list[DeclaredRequirement] | None = None,
labels: Sequence[str] = (),
) -> list[FunctionTool]:
"""The navigator's tools: survey the catalogue, open one base, read one document — and, when
the caller offers the two sinks, DECLARE the requirement that binds a direction.
@ -1465,7 +1516,7 @@ def navigator_tools(
# ``binds`` says out loud what the declaration is for: without it the reply is data with no
# instruction, and the instruction is the whole correction.
declared = _declared_document(index, bundle_id, path)
return {
reply: dict[str, Any] = {
"declared": True,
"bundle_id": bundle_id,
"path": path,
@ -1477,6 +1528,30 @@ def navigator_tools(
"declaration of a requirement that is not about the measure is worth nothing."
),
}
# P22 DEL B: turn the reply into a COMPARISON against what this run was commissioned to
# do. P20/A1 made the reply carry the document's own words; measured, that was not enough
# on its own - three rounds of declarations and not one named a fasit concept. The words
# compared are the DOCUMENT's (read off the base), never ``ref``, which is the caller's own
# argument echoed back: a comparison against the caller's input can only ever agree.
# Absent labels (the exploration mints its own directions, so there are none at
# declaration time) the two keys are omitted and the reply is byte-identical to P20's.
if labels:
overlap = _label_overlap(labels, declared[0], declared[1])
listed = ", ".join(repr(label) for label in labels)
reply["directions"] = list(labels)
reply["overlap"] = list(overlap)
reply["compare"] = (
f"You declared {declared[0]!r} ({declared[1]!r}) for these directions: {listed}. "
+ (
f"Words they share: {', '.join(overlap)}."
if overlap
else "No word of any of them appears in the document's own title or number. "
"If this document is not about the measure you declared it for, it is the "
"wrong requirement: filter the level again with a word from the direction "
"itself and read the candidates that come back."
)
)
return reply
tools = [list_bundles, read_bundle, read_dir, read_file]
if requirements is not None:

View file

@ -1294,6 +1294,12 @@ async def run_project(
dimension=dimension_id,
opened=debate_tool_calls,
requirements=debate_requirements,
# P22 DEL B: the commission's own direction names, so the declaration rung can
# answer with a COMPARISON instead of a confirmation. RUN-level, exactly as
# the declaration is (P19 A4): the debate declares once per run, so the reply
# names every direction the run carries rather than picking one it cannot
# attribute. Without a mandate this is empty and the reply is unchanged.
labels=[a.label for a in mandate.approaches] if mandate else (),
)
)
# What the navigation could NOT reach, taken from the run's ONE walk. The road path below