feat(p22): declare_requirement answers with a COMPARISON, not a confirmation
P19 DEL A made a direction name the requirement that binds it; P20/A1 made the reply carry the DOCUMENT's own title and number instead of echoing the caller's arguments. Re-measured at the head of this session against the six round-5 debate traces: requirement_hit is 0 of 20 approach rows and 0 of 12 declarations -- the third round in a row at zero. P21/C1 made the runs LOOK first and it worked on its own terms (distinct documents before a declaration went 1,1,1,2,5,13 -> 3,3,5,7,11,12) and the hit did not move. The runs were made to read MORE, not righter. The reply now compares: it names the directions the run was commissioned to pursue and says which of their words appear in the declared document's own title and number, or that none do. A REPORT, never a gate -- the declaration is recorded either way, because a requirement can bind a measure without sharing a word with the name someone gave it, which is exactly how the alternative rule the C1 measurement rejected failed one rung over. The words compared are the DOCUMENT's, never `ref`: a comparison against the caller's own argument can only ever agree. Matching is generous in both directions, and that failure direction is chosen -- a false "no overlap" pushes a model away from a declaration that was right, a false "overlap" merely keeps the report quiet. MEASURED BEFORE IT WAS BUILT, offline against the six traces as the order required (no paid calls in DEL B): the rule speaks on 10 of 12 declarations and stays quiet on 2. A rule that spoke on 12 of 12, or on 0 of 12, could not tell the two classes apart. `labels` defaults to empty, so every call site written before today is byte-identical and the three keys are ABSENT rather than empty -- "there was nothing to compare against" and "we compared and found nothing" are different facts. RUN-level, as the declaration itself is (P19 A4). Also re-measured: the order cited requirement_hit as "0 of 12". The field is per APPROACH (0 of 20); 12 is the number of DECLARATIONS (7 distinct, 0 hits). Both zero, so the conclusion stands, but they are two populations. Load-bearing MEASURED (tests/test_requirement_comparison_loadbearing.py, 8 arms), eight mutations all red against the WHOLE suite + green control 1881/5 (from 1873/5, superset, 0 removed) and golden demo-transcript.stdout BYTE-UNCHANGED (shasum -a 1 of the CONTENT = ea8c534773acdbe41ae68f2c55724d69aaf8be4f): B1 detach the run.py wiring (1 red, that arm alone) - B2 always report an overlap (5) - B3 never report one (2) - B4 compare against the caller's ref (1, that arm alone) - B5 make it a gate (5) - B6 emit the keys with no directions (2, one an OLDER independent witness) - B7 exact token equality instead of substring (1) - B8 drop the minimum word length (3). Honesty limits, stated: no LIVE model has read the comparison yet (DEL D is the measurement); the report cannot say a requirement IS right, only that it shares no word with the direction; and finding 4 (`named` 1/20) is this same matter from the other side, so DEL D measures it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
0a81de2d76
commit
9072359606
5 changed files with 360 additions and 1 deletions
43
CLAUDE.md
43
CLAUDE.md
|
|
@ -3011,6 +3011,49 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
|
||||||
uttalt:** ingen betalt kjoering bekrefter ennaa at nekten endrer utfallet levende (DEL D er
|
uttalt:** ingen betalt kjoering bekrefter ennaa at nekten endrer utfallet levende (DEL D er
|
||||||
maalingen); vinduet er aldri naadd av noe maalt, saa avkortnings-grenen er oevd kun syntetisk; og
|
maalingen); vinduet er aldri naadd av noe maalt, saa avkortnings-grenen er oevd kun syntetisk; og
|
||||||
skjemaet naar fortsatt ikke prompten, saa foerste forsoek gjetter som foer.
|
skjemaet naar fortsatt ikke prompten, saa foerste forsoek gjetter som foer.
|
||||||
|
- **`declare_requirement` svarer med en SAMMENLIGNING, ikke en bekreftelse (P22 DEL B, 16.09):**
|
||||||
|
P19 DEL A gjorde at en retning MAA navngi kravet som binder den, og P20/A1 lot svaret baere
|
||||||
|
DOKUMENTETS egen tittel og nummer i stedet for aa ekko kallerens argumenter. MAALT paa nytt ved
|
||||||
|
starten av oekt 126 mot de seks runde-5-sporene: `requirement_hit` er **0 av 20** approach-rader
|
||||||
|
og **0 av 12** erklaeringer — tredje runde paa rad paa null. P21/C1 fikk kjoeringene til aa SE
|
||||||
|
seg om, og virket paa sine egne premisser (distinkte dokumenter foer en erklaering
|
||||||
|
`1·1·1·2·5·13` → `3·3·5·7·11·12`), men treffet rikket seg ikke: kjoeringene ble faatt til aa lese
|
||||||
|
MER, ikke RIKTIGERE. **Ordrens egen denominator var upresis, og det er maalt:** feltet
|
||||||
|
`requirement_hit` er per APPROACH (0 av 20), mens 12 er antallet ERKLAERINGER (7 distinkte, 0
|
||||||
|
treff) — begge null, saa konklusjonen staar, men de er to populasjoner.
|
||||||
|
`_label_overlap` er den ene sammenligningen: hvilke ord fra kommisjonens retninger som finnes i
|
||||||
|
det ERKLAERTE DOKUMENTETS egen tittel og nummer. **En RAPPORT, aldri en gate** — erklaeringen
|
||||||
|
registreres uansett (M B5 → 5 roede): et krav kan binde et tiltak uten aa dele ett ord med navnet
|
||||||
|
noen ga det, hvilket er NOEYAKTIG slik den alternative regelen P21/C1 maalte og forkastet feilet,
|
||||||
|
ett trinn over. **Ordene som sammenlignes er DOKUMENTETS, aldri `ref`** — kallerens eget argument
|
||||||
|
ekkoet tilbake, og en sammenligning mot kallerens input kan bare vaere enig (M B4 → 1 roed, den
|
||||||
|
armen ALENE; P20/A1s regel anvendt paa halvdelen P20 ikke naadde). **Sjenerøs i BEGGE retninger**
|
||||||
|
(delstreng hver vei, saa `rundkjoring` moeter `Rundkjoringer` og `senkekostnader` moeter `senke`),
|
||||||
|
og feilretningen er VALGT: rapporten sier ett av to, og bare ett av dem kan gjoere skade — en
|
||||||
|
falsk «ingen overlapp» skyver en modell BORT fra en erklaering som var riktig, mens en falsk
|
||||||
|
«overlapp» bare gjoer rapporten stille. Delstreng feiler mot stille (P18s `filter` valgte samme
|
||||||
|
retning av samme grunn; M B7 → 1 roed, M B2 → 5, M B3 → 2). `_LABEL_WORD_MIN = 4`, ellers deler
|
||||||
|
hver label «for»/«med»/«til» med et halvt korpus (M B8 → 3 roede).
|
||||||
|
**MAALT FOER DEN BLE BYGGET**, offline mot de seks sporene slik ordren krevde (ingen betalte kall
|
||||||
|
i DEL B): regelen TALER paa **10 av 12** erklaeringer og tier paa 2 (begge fv412, paa
|
||||||
|
`materialer`). En regel som talte paa 12 av 12 — eller paa 0 av 12 — kunne ikke skilt de to
|
||||||
|
klassene, samme proeve P21/C1s terskel maatte bestaa. **`labels` DEFAULTER til tom**, saa hvert
|
||||||
|
kallsted skrevet foer i dag er BYTE-IDENTISK og de tre noeklene UTELATES (fravaerende, ikke tomme:
|
||||||
|
«det fantes ingenting aa sammenligne mot» og «vi sammenlignet og fant ingenting» er ulike fakta,
|
||||||
|
og bare ett av dem er sant der; M B6 → 2 roede, hvorav ett ELDRE uavhengig vitne i
|
||||||
|
`test_binding_requirement`). Utforskningen mynter sine egne retninger, saa den HAR ingen ved
|
||||||
|
erklaerings-tid. **RUN-nivaa, som erklaeringen selv (P19 A4):** debatten erklaerer ÉN gang per
|
||||||
|
kjoering, saa svaret navngir HVER retning kjoeringen baerer i stedet for aa velge én den ikke kan
|
||||||
|
tilskrive. Wiringen i `run.py` maales ATFERDSMESSIG — en kilde-assert er en lint (oekt 77s funn),
|
||||||
|
saa armen driver den EKTE debatten med et steg-manus som erklaerer og leser sammenligningen ut av
|
||||||
|
verktoeyets eget svar (M B1 → 1 roed, den armen ALENE). Load-bearing MAALT
|
||||||
|
(`tests/test_requirement_comparison_loadbearing.py`, 8 armer), **aatte mutasjoner alle roede mot
|
||||||
|
HELE suiten** + groenn kontroll **1881/5** (fra 1873/5, supersett, 0 fjernet) og golden
|
||||||
|
BYTE-UENDRET (`shasum -a 1` av INNHOLDET = `ea8c534773acdbe41ae68f2c55724d69aaf8be4f`).
|
||||||
|
**AErlighets-grenser, uttalt:** ingen LEVENDE modell har lest sammenligningen
|
||||||
|
(structured-output-grensens klasse) — DEL D er maalingen; rapporten kan ikke si at et krav ER
|
||||||
|
riktig, bare at det ikke deler ett ord med retningen; og funn 4 (`named` 1/20) er denne saken
|
||||||
|
sett fra den andre siden, saa den maales av DEL D og ikke av en egen soem.
|
||||||
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
||||||
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -508,6 +508,15 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
||||||
requirement can be seen to be wrong. The route is named in the instruction as well as the
|
requirement can be seen to be wrong. The route is named in the instruction as well as the
|
||||||
answer: pass `read_dir` a `filter` word from the approach's own label.
|
answer: pass `read_dir` a `filter` word from the approach's own label.
|
||||||
|
|
||||||
|
On a commissioned run the reply goes one step further and **compares**: it names the directions
|
||||||
|
the run was commissioned to pursue and says which of their words appear in the declared
|
||||||
|
document's own title and number — or that none do. It is a report, never a gate: the declaration
|
||||||
|
is recorded either way, because a requirement can bind a measure without sharing a word with the
|
||||||
|
name someone gave it. (Measured over three stress rounds, not one declaration named a concept the
|
||||||
|
answer key asks for, and making the runs read more did not change that. The rule was replayed
|
||||||
|
against those traces before it was built: it speaks on 10 of 12 declarations and stays quiet on
|
||||||
|
2.)
|
||||||
|
|
||||||
**What the commissioner counts as success reaches the readers.** A mandate's `success_criteria`
|
**What the commissioner counts as success reaches the readers.** A mandate's `success_criteria`
|
||||||
used to reach the announcement and nothing else. It is now restated verbatim in the debate's task
|
used to reach the announcement and nothing else. It is now restated verbatim in the debate's task
|
||||||
message — the prompt where `declare_requirement` is available — through one renderer, and is
|
message — the prompt where `declare_requirement` is available — through one renderer, and is
|
||||||
|
|
|
||||||
|
|
@ -24,6 +24,7 @@ This module imports ``agent_framework.orchestrations`` and therefore may never b
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
|
import re
|
||||||
import sys
|
import sys
|
||||||
from collections.abc import Awaitable, Callable, Mapping, Sequence
|
from collections.abc import Awaitable, Callable, Mapping, Sequence
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
|
|
@ -1126,12 +1127,62 @@ def _refused_text(exc: Exception) -> str:
|
||||||
return f"REFUSED ({_refusal_kind(exc)}): {exc}"
|
return f"REFUSED ({_refusal_kind(exc)}): {exc}"
|
||||||
|
|
||||||
|
|
||||||
|
#: P22 DEL B - the shortest word of a direction's label that can carry meaning into a comparison.
|
||||||
|
#: Below this every label shares "for", "med", "ny" with half a corpus and the report would speak
|
||||||
|
#: of an overlap nobody meant.
|
||||||
|
_LABEL_WORD_MIN: Final = 4
|
||||||
|
|
||||||
|
|
||||||
|
def _label_overlap(labels: Sequence[str], *document_text: str) -> tuple[str, ...]:
|
||||||
|
"""Which words of the commission's directions appear in the DECLARED DOCUMENT's own text.
|
||||||
|
|
||||||
|
The measured defect (P21 funn 2, re-measured at the head of okt 126): ``requirement_hit`` is
|
||||||
|
**0 of 20** approach rows over round 5's six paid runs and **0 of 12** declarations - a third
|
||||||
|
round in a row - and P21/C1 moved the documents read before a declaration from 1,1,1,2,5,13 to
|
||||||
|
3,3,5,7,11,12 without moving the hit. The runs were made to read MORE, not righter. P20/A1
|
||||||
|
already gives back the document's own title and number; what nobody said was whether that
|
||||||
|
document has anything to do with the direction the run is committed to.
|
||||||
|
|
||||||
|
**A REPORT, never a gate.** The declaration is recorded either way. A gate on word overlap
|
||||||
|
would refuse legitimate declarations - a requirement can bind a measure without sharing a word
|
||||||
|
with the name someone gave it - which is the P21/C1 alternative rule's failure, one rung over.
|
||||||
|
|
||||||
|
**Generous in BOTH directions, and that is the failure direction chosen on purpose.** A token
|
||||||
|
matches when it is a substring of a document token or the document token is a substring of it,
|
||||||
|
so ``rundkjoring`` meets ``Rundkjoringer`` and ``asfaltdekke`` meets ``Asfalt``. The report
|
||||||
|
says one of two things, and only one of them can do harm: a false "no overlap" pushes a model
|
||||||
|
away from a declaration that was right, while a false "overlap" merely keeps the report quiet.
|
||||||
|
Substring matching fails towards quiet. (P18's ``filter`` chose the same direction for the same
|
||||||
|
reason: a substring fails towards showing MORE, which can be narrowed.)
|
||||||
|
|
||||||
|
MEASURED offline against the six round-5 debate traces before this was built: the rule speaks
|
||||||
|
on **10 of 12** declarations and stays quiet on 2 (both fv412, on ``materialer``). A rule that
|
||||||
|
spoke on 12 of 12 or on 0 of 12 could not tell the two classes apart, which is the same test
|
||||||
|
P21/C1's threshold had to pass.
|
||||||
|
"""
|
||||||
|
haystack = {
|
||||||
|
token
|
||||||
|
for text in document_text
|
||||||
|
for token in re.split(r"[\W_]+", text.lower())
|
||||||
|
if len(token) >= _LABEL_WORD_MIN
|
||||||
|
}
|
||||||
|
shared: set[str] = set()
|
||||||
|
for label in labels:
|
||||||
|
for token in re.split(r"[\W_]+", label.lower()):
|
||||||
|
if len(token) < _LABEL_WORD_MIN:
|
||||||
|
continue
|
||||||
|
if any(token in other or other in token for other in haystack):
|
||||||
|
shared.add(token)
|
||||||
|
return tuple(sorted(shared))
|
||||||
|
|
||||||
|
|
||||||
def navigator_tools(
|
def navigator_tools(
|
||||||
bundle_dirs: Sequence[str],
|
bundle_dirs: Sequence[str],
|
||||||
*,
|
*,
|
||||||
dimension: str | None = None,
|
dimension: str | None = None,
|
||||||
opened: list[ToolCall] | None = None,
|
opened: list[ToolCall] | None = None,
|
||||||
requirements: list[DeclaredRequirement] | None = None,
|
requirements: list[DeclaredRequirement] | None = None,
|
||||||
|
labels: Sequence[str] = (),
|
||||||
) -> list[FunctionTool]:
|
) -> list[FunctionTool]:
|
||||||
"""The navigator's tools: survey the catalogue, open one base, read one document — and, when
|
"""The navigator's tools: survey the catalogue, open one base, read one document — and, when
|
||||||
the caller offers the two sinks, DECLARE the requirement that binds a direction.
|
the caller offers the two sinks, DECLARE the requirement that binds a direction.
|
||||||
|
|
@ -1465,7 +1516,7 @@ def navigator_tools(
|
||||||
# ``binds`` says out loud what the declaration is for: without it the reply is data with no
|
# ``binds`` says out loud what the declaration is for: without it the reply is data with no
|
||||||
# instruction, and the instruction is the whole correction.
|
# instruction, and the instruction is the whole correction.
|
||||||
declared = _declared_document(index, bundle_id, path)
|
declared = _declared_document(index, bundle_id, path)
|
||||||
return {
|
reply: dict[str, Any] = {
|
||||||
"declared": True,
|
"declared": True,
|
||||||
"bundle_id": bundle_id,
|
"bundle_id": bundle_id,
|
||||||
"path": path,
|
"path": path,
|
||||||
|
|
@ -1477,6 +1528,30 @@ def navigator_tools(
|
||||||
"declaration of a requirement that is not about the measure is worth nothing."
|
"declaration of a requirement that is not about the measure is worth nothing."
|
||||||
),
|
),
|
||||||
}
|
}
|
||||||
|
# P22 DEL B: turn the reply into a COMPARISON against what this run was commissioned to
|
||||||
|
# do. P20/A1 made the reply carry the document's own words; measured, that was not enough
|
||||||
|
# on its own - three rounds of declarations and not one named a fasit concept. The words
|
||||||
|
# compared are the DOCUMENT's (read off the base), never ``ref``, which is the caller's own
|
||||||
|
# argument echoed back: a comparison against the caller's input can only ever agree.
|
||||||
|
# Absent labels (the exploration mints its own directions, so there are none at
|
||||||
|
# declaration time) the two keys are omitted and the reply is byte-identical to P20's.
|
||||||
|
if labels:
|
||||||
|
overlap = _label_overlap(labels, declared[0], declared[1])
|
||||||
|
listed = ", ".join(repr(label) for label in labels)
|
||||||
|
reply["directions"] = list(labels)
|
||||||
|
reply["overlap"] = list(overlap)
|
||||||
|
reply["compare"] = (
|
||||||
|
f"You declared {declared[0]!r} ({declared[1]!r}) for these directions: {listed}. "
|
||||||
|
+ (
|
||||||
|
f"Words they share: {', '.join(overlap)}."
|
||||||
|
if overlap
|
||||||
|
else "No word of any of them appears in the document's own title or number. "
|
||||||
|
"If this document is not about the measure you declared it for, it is the "
|
||||||
|
"wrong requirement: filter the level again with a word from the direction "
|
||||||
|
"itself and read the candidates that come back."
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return reply
|
||||||
|
|
||||||
tools = [list_bundles, read_bundle, read_dir, read_file]
|
tools = [list_bundles, read_bundle, read_dir, read_file]
|
||||||
if requirements is not None:
|
if requirements is not None:
|
||||||
|
|
|
||||||
|
|
@ -1294,6 +1294,12 @@ async def run_project(
|
||||||
dimension=dimension_id,
|
dimension=dimension_id,
|
||||||
opened=debate_tool_calls,
|
opened=debate_tool_calls,
|
||||||
requirements=debate_requirements,
|
requirements=debate_requirements,
|
||||||
|
# P22 DEL B: the commission's own direction names, so the declaration rung can
|
||||||
|
# answer with a COMPARISON instead of a confirmation. RUN-level, exactly as
|
||||||
|
# the declaration is (P19 A4): the debate declares once per run, so the reply
|
||||||
|
# names every direction the run carries rather than picking one it cannot
|
||||||
|
# attribute. Without a mandate this is empty and the reply is unchanged.
|
||||||
|
labels=[a.label for a in mandate.approaches] if mandate else (),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
# What the navigation could NOT reach, taken from the run's ONE walk. The road path below
|
# What the navigation could NOT reach, taken from the run's ONE walk. The road path below
|
||||||
|
|
|
||||||
226
tests/test_requirement_comparison_loadbearing.py
Normal file
226
tests/test_requirement_comparison_loadbearing.py
Normal file
|
|
@ -0,0 +1,226 @@
|
||||||
|
"""P22 DEL B — ``declare_requirement`` answers with a COMPARISON, not a confirmation.
|
||||||
|
|
||||||
|
MEASURED (P21 funn 2, re-measured at the head of okt 126 against the six round-5 debate traces):
|
||||||
|
``requirement_hit`` is **0 of 20** approach rows and **0 of 12** declarations — the third round in
|
||||||
|
a row at zero. P21/C1 made the runs LOOK first, and it worked on its own terms: distinct documents
|
||||||
|
opened before a declaration went from 1,1,1,2,5,13 to 3,3,5,7,11,12. The hit did not move. The
|
||||||
|
runs were made to read more, not righter.
|
||||||
|
|
||||||
|
P20/A1 had already made the reply carry the document's OWN title and number instead of echoing the
|
||||||
|
caller's arguments. What nobody said was whether that document has anything to do with the
|
||||||
|
direction the run is committed to. This turns the reply into that comparison.
|
||||||
|
|
||||||
|
**A REPORT, never a gate.** The declaration is recorded either way, and arm (c) is what holds that
|
||||||
|
open. A gate on word overlap would refuse legitimate declarations — a requirement can bind a
|
||||||
|
measure without sharing a word with the name someone gave it — which is precisely how the
|
||||||
|
alternative rule P21/C1 measured and rejected failed, one rung over.
|
||||||
|
|
||||||
|
**Measured before it was built**, offline against the six traces: the rule speaks on **10 of 12**
|
||||||
|
declarations and stays quiet on 2 (both fv412, on ``materialer``). A rule that spoke on 12 of 12,
|
||||||
|
or on 0 of 12, could not tell the two classes apart.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from portfolio_optimiser.simulation import scripted_factory
|
||||||
|
|
||||||
|
from portfolio_optimiser import okf
|
||||||
|
from portfolio_optimiser.explore import ToolCall, navigator_tools
|
||||||
|
from portfolio_optimiser.mandate import Approach, Mandate
|
||||||
|
from portfolio_optimiser.run import run_project
|
||||||
|
|
||||||
|
_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
|
||||||
|
_TUNNEL = _EXAMPLES / "tunnel-hauglia"
|
||||||
|
_BASE_ID = "tunnel-hauglia"
|
||||||
|
|
||||||
|
#: The fixture document the arms declare. Its title is ASCII-clean, which is what lets a label
|
||||||
|
#: share a word with it without a marker carrying a multibyte character into a scripted run.
|
||||||
|
_DOC = "tiltak-portalskjerming.md"
|
||||||
|
|
||||||
|
#: A direction whose words are IN that title ("Portalskjerming: senke L20 ...").
|
||||||
|
_MATCHING = "Billigere portalskjerming"
|
||||||
|
#: A direction that shares nothing with it. Checked against the document's own tokens, both ways.
|
||||||
|
_FOREIGN = "Asfaltdekke gjenbruk"
|
||||||
|
|
||||||
|
|
||||||
|
def _wired(labels: tuple[str, ...]) -> tuple[dict[str, Any], list[ToolCall], list[Any]]:
|
||||||
|
opened: list[ToolCall] = []
|
||||||
|
declared: list[Any] = []
|
||||||
|
tools = navigator_tools((str(_TUNNEL),), opened=opened, requirements=declared, labels=labels)
|
||||||
|
return {t.name: t for t in tools}, opened, declared
|
||||||
|
|
||||||
|
|
||||||
|
def _declare(
|
||||||
|
labels: tuple[str, ...], *, ref: str = "Krav 1.1-1"
|
||||||
|
) -> tuple[dict[str, Any], list[Any]]:
|
||||||
|
tools, opened, declared = _wired(labels)
|
||||||
|
for name in [f.name for f in okf.navigate_bundle(str(_TUNNEL)).context_files][:3]:
|
||||||
|
opened.append(ToolCall(name="read_file", bundle_id=_BASE_ID, path=name))
|
||||||
|
opened.append(ToolCall(name="read_file", bundle_id=_BASE_ID, path=_DOC))
|
||||||
|
answer = tools["declare_requirement"].func(bundle_id=_BASE_ID, path=_DOC, ref=ref)
|
||||||
|
return answer, declared
|
||||||
|
|
||||||
|
|
||||||
|
# --- (a)/(b) the discriminator: SAME document, SAME ref, different directions ---------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_direction_that_shares_a_word_is_told_which_one() -> None:
|
||||||
|
"""KNOWN-POSITIVE. The document's title carries the direction's own word, and the reply says
|
||||||
|
so — the half that keeps the report from being one that only ever complains."""
|
||||||
|
answer, _ = _declare((_MATCHING,))
|
||||||
|
assert answer["overlap"] == ["portalskjerming"], answer["overlap"]
|
||||||
|
assert "portalskjerming" in answer["compare"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_direction_that_shares_nothing_is_told_that_too() -> None:
|
||||||
|
"""KNOWN-NEGATIVE, and the measured case: the SAME document and the SAME ``ref`` as the arm
|
||||||
|
above — the only difference is the direction. That is what makes this a comparison rather than
|
||||||
|
a confirmation, and it is exactly the 10-of-12 class the round-5 traces fall into."""
|
||||||
|
answer, _ = _declare((_FOREIGN,))
|
||||||
|
assert answer["overlap"] == []
|
||||||
|
assert "No word of any of them appears" in answer["compare"], answer["compare"]
|
||||||
|
assert repr(_FOREIGN) in answer["compare"], answer["compare"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- (c) a REPORT, never a gate ------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_declaration_with_no_overlap_is_still_recorded() -> None:
|
||||||
|
"""The whole difference between this rung and P21/C1's. A requirement can bind a measure
|
||||||
|
without sharing a word with the name someone gave it, so refusing here would refuse
|
||||||
|
legitimate declarations — the failure of the alternative rule the C1 measurement rejected."""
|
||||||
|
answer, declared = _declare((_FOREIGN,))
|
||||||
|
assert answer["declared"] is True
|
||||||
|
assert len(declared) == 1, "a reported mismatch must not swallow the declaration"
|
||||||
|
|
||||||
|
|
||||||
|
# --- (d) it reads the DOCUMENT, never the caller's own argument ----------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_comparison_never_reads_the_callers_own_ref() -> None:
|
||||||
|
"""``ref`` is the caller's argument echoed back, and a comparison against the caller's own
|
||||||
|
input can only ever agree — P20/A1's rule (read off the base, never off the arguments)
|
||||||
|
applied to the half P20 did not reach. A ``ref`` stuffed with the direction's words must not
|
||||||
|
manufacture an overlap."""
|
||||||
|
answer, _ = _declare((_FOREIGN,), ref="Asfaltdekke gjenbruk krav")
|
||||||
|
assert answer["overlap"] == [], answer["overlap"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- (e) no directions -> the P20 reply, unchanged ------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_without_directions_the_reply_is_the_one_p20_shipped() -> None:
|
||||||
|
"""The exploration mints its own directions, so at declaration time it HAS none — and every
|
||||||
|
call site that passes no labels must be byte-identical to before. Absent keys, not empty ones:
|
||||||
|
"there was nothing to compare against" and "we compared and found nothing" are different
|
||||||
|
facts, and only one of them is true here."""
|
||||||
|
answer, _ = _declare(())
|
||||||
|
assert set(answer) == {
|
||||||
|
"declared",
|
||||||
|
"bundle_id",
|
||||||
|
"path",
|
||||||
|
"ref",
|
||||||
|
"title",
|
||||||
|
"req_number",
|
||||||
|
"binds",
|
||||||
|
}, sorted(answer)
|
||||||
|
|
||||||
|
|
||||||
|
# --- (f)/(g) the matching rule's two measured properties -----------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_matching_is_generous_in_both_directions() -> None:
|
||||||
|
"""The failure direction chosen on purpose. A direction whose word is a PREFIX of the
|
||||||
|
document's own word counts, and so does the reverse — Norwegian inflects ("rundkjoring" vs
|
||||||
|
"Rundkjoringer") and a strict rule would report "no overlap" on a declaration that was right,
|
||||||
|
which is the only one of the two errors that can push a model away from a correct answer."""
|
||||||
|
# label word LONGER than the document's own ("portalskjerming" is a prefix of it)
|
||||||
|
assert _declare(("Portalskjermingen paa nordsiden",))[0]["overlap"] == ["portalskjermingen"]
|
||||||
|
# and SHORTER: the document says "senke", the direction "senkekostnader"
|
||||||
|
assert _declare(("Senkekostnader",))[0]["overlap"] == ["senkekostnader"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_short_words_cannot_manufacture_an_overlap() -> None:
|
||||||
|
"""Below the minimum every label shares "for", "med", "til" with half a corpus, and the report
|
||||||
|
would speak of an overlap nobody meant. The control is the arm above: the same rule that
|
||||||
|
rejects these still finds the real word."""
|
||||||
|
answer, _ = _declare(("Ny og med til for",))
|
||||||
|
assert answer["overlap"] == [], answer["overlap"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- (h) the commission's labels actually REACH the tool in a run --------------------------------
|
||||||
|
|
||||||
|
_PID = "TUNNEL-HAUGLIA"
|
||||||
|
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (test)"}
|
||||||
|
_VALID_REPLY = (
|
||||||
|
'{"measure":"LED-retrofit","affected_items":'
|
||||||
|
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],'
|
||||||
|
'"claimed_saving_nok":30000}'
|
||||||
|
)
|
||||||
|
_CHECKER_REPLY = "Reasoning holds.\nVERDICT: APPROVE"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_commissioned_run_reaches_the_tool_with_its_own_directions() -> None:
|
||||||
|
"""The WIRING, measured behaviourally rather than by reading ``run.py``. A source assertion is
|
||||||
|
a lint; this drives the real debate with a step manuscript that declares, and reads the
|
||||||
|
comparison back out of the tool's own answer. Without ``labels=`` in ``run.py`` the reply
|
||||||
|
carries no ``compare`` at all and this arm falls."""
|
||||||
|
concepts = [f.name for f in okf.navigate_bundle(str(_TUNNEL)).context_files]
|
||||||
|
script = {
|
||||||
|
"proposer": [
|
||||||
|
*(
|
||||||
|
{"call": "read_file", "args": {"bundle_id": _BASE_ID, "path": name}}
|
||||||
|
for name in concepts[:3]
|
||||||
|
),
|
||||||
|
{"call": "read_file", "args": {"bundle_id": _BASE_ID, "path": _DOC}},
|
||||||
|
{
|
||||||
|
"call": "declare_requirement",
|
||||||
|
"args": {"bundle_id": _BASE_ID, "path": _DOC, "ref": "Krav 1.1-1"},
|
||||||
|
},
|
||||||
|
_VALID_REPLY,
|
||||||
|
_VALID_REPLY,
|
||||||
|
_VALID_REPLY,
|
||||||
|
_VALID_REPLY,
|
||||||
|
],
|
||||||
|
"checker": _CHECKER_REPLY,
|
||||||
|
}
|
||||||
|
seen: list[str] = []
|
||||||
|
|
||||||
|
def factory(role: str) -> Any:
|
||||||
|
client = scripted_factory(script, [])(role)
|
||||||
|
original = client._inner_get_response
|
||||||
|
|
||||||
|
def recording(*, messages, options, stream=False, **kwargs): # type: ignore[no-untyped-def]
|
||||||
|
for message in messages:
|
||||||
|
for content in getattr(message, "contents", ()) or ():
|
||||||
|
value = getattr(content, "result", None)
|
||||||
|
if value:
|
||||||
|
seen.append(str(value))
|
||||||
|
return original(messages=messages, options=options, stream=stream, **kwargs)
|
||||||
|
|
||||||
|
client._inner_get_response = recording
|
||||||
|
return client
|
||||||
|
|
||||||
|
await run_project(
|
||||||
|
_PID,
|
||||||
|
"local",
|
||||||
|
docs_dir=str(_TUNNEL),
|
||||||
|
bundle_dir=str(_TUNNEL),
|
||||||
|
verdict_input=_VERDICT_INPUT,
|
||||||
|
client_factory=factory,
|
||||||
|
mandate=Mandate(
|
||||||
|
objective="Find savings",
|
||||||
|
approaches=(Approach(id="a1", label=_FOREIGN, description="expert's reason"),),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
blob = "\n".join(seen)
|
||||||
|
assert "No word of any of them appears" in blob, (
|
||||||
|
"the commission's directions never reached the declaration rung: " + blob[-600:]
|
||||||
|
)
|
||||||
|
assert _FOREIGN in blob
|
||||||
Loading…
Add table
Add a link
Reference in a new issue