feat(p21): a declaration that must have LOOKED, and a refusal that names the neighbours
C1. Round 4 produced 13 declarations over six runs and NOT ONE named a fasit concept. The distinct documents opened before each were 1,1,1,1,1,1,1,2,5,5,6,13,13: seven declared the base's FIRST requirement after opening exactly ONE document. The order offered two rules and asked which discriminates. Replayed against the real listings: "the declared document must have come back from a read_dir filtered on a word from the approach's label" refuses 13 of 13 -- including Soraasen's 12.11, the closest any run came -- because ZERO of the 13 were reached through a filtered listing at all. A gate that refuses every measured case, right and wrong alike, cannot discriminate. "fewer than k distinct documents opened" at k=3 refuses 8 of 13 and keeps the five that navigated. k=3, 4 and 5 refuse the SAME eight -- the distribution has a gap between 2 and 5 -- so the threshold is not on a cliff, and 3 is the lowest of that plateau. DISTINCT paths, not calls, and capped by the base's own size so a small base stays declarable. C2. Over the same traces 18 of 143 path-bearing calls named a path the base does not hold, ELEVEN of them one run walking R761/4-3, 4.3, 4-2, 4-1, 4-0, 4-5, 4-6 while the real names are R761/4, R761/41, R761/42. The refusal already named the nearest listable ancestor; now it also names up to five of that rung's own subdirectories, ranked by longest common prefix with the segment that failed. ONE copy shared by both refusal sites, built from context_files through in_dimension, so every name handed back resolves and the verdict layer can never be advertised in an apology. MEASURED after: 16 of 18. Load-bearing MEASURED, four mutations all red against the WHOLE suite, green control 1863/5, golden byte-unchanged: C3(i) the declaration gate detached (2 red) . C3(ii) the neighbour list empty (6) . C3(iii) built from files (1, the verdict arm alone) . C3(iv) count CALLS instead of distinct documents (1, the repetition arm alone). Three existing arms REWRITTEN, not weakened: all three read one document and declared, which is the measured failure class exactly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
7b4f85d77c
commit
ac0bfdba27
8 changed files with 534 additions and 28 deletions
53
CLAUDE.md
53
CLAUDE.md
|
|
@ -2918,6 +2918,59 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
|
|||
halvdeler står); portefølje-armen er ikke wiret (flagget er nektet der ved navn, så en utskrift
|
||||
ville vært død kode); og alle beløp i de fem skjemaene er OPPDIKTEDE størrelsesordener, som hvert
|
||||
setts eget `honesty`-felt sier.
|
||||
- **En erklæring må ha LETT, og en nekt navngir naboene — ordrens to kandidat-regler ble skilt av
|
||||
MÅLING (P21 DEL C, 15.09):** P19 DEL A gjorde at en retning MÅ navngi kravet som binder den, og
|
||||
rungen virker — runde 4 ga 13 erklæringer over seks kjøringer. **Ikke én navnga et
|
||||
fasit-konsept**, og distinkte dokumenter åpnet før hver av dem var
|
||||
`1,1,1,1,1,1,1,2,5,5,6,13,13`: sju erklærte basens FØRSTE krav etter å ha åpnet ETT dokument.
|
||||
**C1 — ordren ba om en måling, ikke en preferanse, og målingen valgte:** alternativet («det
|
||||
erklærte dokumentet må ha blitt returnert av et `read_dir` filtrert på et ord fra approachens
|
||||
label») ble spilt av mot de EKTE listingene og nekter **13 av 13** — inkludert sorasens `12.11`,
|
||||
som ordren navngir som det nærmeste noen kjøring kom; MÅLT kom **null** av de 13 erklæringene
|
||||
gjennom en filtrert listing i det hele tatt. En gate som nekter hvert målte tilfelle, riktige som
|
||||
gale, kan ikke SKILLE — det er vakuøs-gate-klassens speilbilde. Den andre regelen (**færre enn
|
||||
*k* distinkte dokumenter åpnet**) nekter ved **k = 3** **8 av 13** og beholder de fem som
|
||||
navigerte, `12.11` blant dem. **Terskelen står ikke på en klippekant:** k = 3, 4 og 5 nekter
|
||||
NØYAKTIG de samme åtte, fordi fordelingen har et gap mellom 2 og 5 — 3 er laveste rad i platået,
|
||||
altså det minste som skiller de to målte klassene. **DISTINKTE stier, ikke kall** (å lese samme
|
||||
dokument tre ganger er ikke navigasjon), og **CAPPET av basens egen størrelse**
|
||||
(`min(3, len(context_files))`) — et fast gulv over en liten base ville gjort erklæring UMULIG
|
||||
der, altså en gate som bare kan nekte, på nøyaktig de små fixturene repoet er bygget på. Nevneren
|
||||
står i nekten («1 distinct document(s) of the 5»), fordi paret er diagnosen (kø-(y) ett hakk ned),
|
||||
og P19-gaten («du leste den aldri») er URØRT og sjekkes FØRST: de to er ulike fakta, og den
|
||||
første kan rettes med ett kall. **C2 — nekten navngir naboene:** over de samme seks sporene navnga
|
||||
**18 av 143** sti-bærende verktøykall en sti basen ikke holder, og **ELLEVE** av dem er ÉN kjøring
|
||||
som vandrer `R761/4-3`, `4.3`, `4-2`, `4-1`, `4-0`, `4-5`, `4-6` — gjetting på en
|
||||
kapittelnummer-skrivemåte korpuset ikke bruker, mens de ekte navnene er `R761/4`, `R761/41`,
|
||||
`R761/42`. Nekten navnga alt den nærmeste LISTBARE forfaren, som er riktig rung; det den ikke
|
||||
kunne si var hvilket av den rungens navn som var ment. `okf.nearest_subdirectories` +
|
||||
`nearest_listable_directory` er ÉN kopi delt av BEGGE nekt-stedene (`read_file` i `explore.py` og
|
||||
`directory_listing` i `okf.py`) — ett spørsmål om én base må ikke ha to svar (kø-(p)) — og
|
||||
`explore._nearest_listable` er FOLDET INN i dem i stedet for å stå som en andre kopi.
|
||||
**Bygget fra `context_files`, ALDRI `files`, og gjennom SAMME `in_dimension`-predikat listingen
|
||||
bruker:** et forslag lest av filsystemet kunne navngitt en katalog `read_dir` så nekter, og ett
|
||||
bygget fra `files` kunne navngitt `type: verdict`-laget VED STI — å reklamere i en NEKT for det
|
||||
ene laget ingen listing nevner er den samme lekkasjen i unnskyldningens klær. **Rangert etter
|
||||
lengste felles prefiks med segmentet som feilet**, så kortest, så navn — dét er hva som setter
|
||||
`R761/4` FØRST for `4-3`; uten felles prefiks i det hele tatt degraderer ordenen til «de korteste
|
||||
navnene på dette nivået», som er et ærlig «her er hva som ER her». En rangering kan ikke nekte
|
||||
noe (dette er hjelpetekst på en nekt), så feilretningen er godartet. **MÅLT ETTER: 16 av 18**
|
||||
nekter navngir nå minst én nabo; de to som ikke gjør det har en forfar som holder dokumenter og
|
||||
ingen underkataloger, og der UTELATES klausulen (omisjon, aldri en setning med ingenting i).
|
||||
Load-bearing MÅLT (`tests/test_declaration_and_neighbours_loadbearing.py`, 13 armer), **fire
|
||||
mutasjoner alle røde mot HELE suiten** + grønn kontroll **1863/5** og golden BYTE-UENDRET
|
||||
(`shasum -a 1` av INNHOLDET = `ea8c534773acdbe41ae68f2c55724d69aaf8be4f`): C3(i) erklærings-gaten
|
||||
detachet (2 røde) · C3(ii) nabolista tom (6) · C3(iii) bygget fra `files` (1, verdict-armen
|
||||
ALENE) · C3(iv) tell KALL i stedet for distinkte dokumenter (1, gjentakelses-armen ALENE).
|
||||
**TRE EKSISTERENDE ARMER ER SKREVET OM, IKKE SVEKKET** (`test_binding_requirement`s
|
||||
korreksjons-arm, `test_right_requirement`s tunnel-arm og `test_across_bundles_cli`s
|
||||
per-base-sink-arm): alle tre leste ETT dokument og erklærte, altså nøyaktig den målte
|
||||
feilklassen; de leser nå tre, og armene beholder sin betydning — en erklæring kjøringens EGET
|
||||
spor støtter blir AKSEPTERT og REGISTRERT. **Ærlighets-grenser, uttalt:** sporet registrerer ikke
|
||||
om en `read_file` LYKTES (recorderen appender FØR `call_next`, funn-99-formen), så tre nektede
|
||||
lesninger teller mot gulvet — regelen måler at kjøringen SÅ seg om, ikke at den forsto; ingen
|
||||
LEVENDE modell har møtt noen av de to nektene (structured-output-grensens klasse); og C2 er
|
||||
hjelpetekst, ikke en gate — den kan ikke gjøre en gjettet sti riktig, bare billigere å rette.
|
||||
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
||||
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
||||
|
||||
|
|
|
|||
|
|
@ -777,6 +777,14 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
|||
run also writes `{run_id}-exploration.json`: the per-round ledger, the plan reviews and the
|
||||
in-loop advisory verdicts, written even when a cap cut the exploration short.
|
||||
|
||||
A declaration of the binding requirement is refused when the run has opened fewer than three
|
||||
distinct documents of the base (capped by the base's own size) — measured, seven of round 4's
|
||||
thirteen declarations named the base's FIRST requirement after opening exactly one document, and
|
||||
none of the thirteen named a right one. The refusal is a turn the model can correct, and it
|
||||
carries both numbers. A path the base does not hold is likewise refused by name, and the refusal
|
||||
now lists up to five subdirectories of the nearest directory that does hold documents — every one
|
||||
of them a path `read_dir` will answer for.
|
||||
|
||||
Every tool call recorded in `{run_id}-exploration.json` and `{run_id}-debate.json` says HOW the
|
||||
level was asked for — `filter`, `offset` and `limit` beside the tool name, the base and the path
|
||||
— so "did the model narrow the level, or page through it" is readable from the artefact rather
|
||||
|
|
|
|||
|
|
@ -1025,6 +1025,13 @@ def _declared_document(index: Mapping[str, str], bundle_id: str, path: str) -> t
|
|||
return "", ""
|
||||
|
||||
|
||||
#: P21/C1 — how many DISTINCT documents a run must have opened before a declaration of the binding
|
||||
#: requirement is worth recording. MEASURED, never chosen: see ``_declare_requirement`` for the
|
||||
#: full distribution and for why the alternative rule the order offered was rejected. Capped by the
|
||||
#: base's own document count at the call site, so a small base stays declarable.
|
||||
_MIN_DOCUMENTS_READ: Final = 3
|
||||
|
||||
|
||||
#: Characters of the root index body one catalogue entry may carry. The catalogue's job is to let a
|
||||
#: manager pick a base, not to read one, so the excerpt is a fixed-size window rather than a share
|
||||
#: of the base: cost then scales with how many bases are configured, which the operator chose, and
|
||||
|
|
@ -1079,26 +1086,19 @@ def _refusal_kind(exc: Exception) -> str:
|
|||
return type(exc).__name__
|
||||
|
||||
|
||||
def _nearest_listable(bundle_dir: str, path: str, dimension: str | None) -> str:
|
||||
"""The deepest ANCESTOR of ``path`` that ``read_dir`` will actually answer for.
|
||||
def _neighbours(bundle_dir: str, path: str, dimension: str | None) -> tuple[str, tuple[str, ...]]:
|
||||
"""``(nearest listable ancestor, up to five of its subdirectories)`` for a path that is absent.
|
||||
|
||||
Chosen off the NAVIGATED ``context_files`` rather than off the filesystem, for two reasons that
|
||||
are the same reason: a directory can exist on disk and hold no navigated concept (nothing links
|
||||
it), in which case ``read_dir`` refuses it and the refusal would have handed the caller a path
|
||||
that does not resolve — ``_index_excerpt``'s rule one rung up, a path that never was is worse
|
||||
than no path. And ``context_files`` is the property that drops the ``type: verdict`` layer, so a
|
||||
refusal can never advertise by name the one layer no listing mentions.
|
||||
|
||||
Falls back to ``""``, the base's own top level, which ``directory_listing`` always answers.
|
||||
ONE navigation for both halves, and BOTH read off ``okf`` rather than reimplemented here: this
|
||||
module and ``directory_listing``'s own refusal answer the same question about the same base, and
|
||||
two copies of "which directory did they mean" would be free to give a caller two answers about
|
||||
one level (kø-(p)).
|
||||
"""
|
||||
bundle = okf.navigate_bundle(bundle_dir)
|
||||
reachable = [f for f in bundle.context_files if okf.in_dimension(f, dimension)]
|
||||
segments = path.strip("/").split("/")
|
||||
for depth in range(len(segments) - 1, 0, -1):
|
||||
candidate = "/".join(segments[:depth])
|
||||
if any(f.name.startswith(candidate + "/") for f in reachable):
|
||||
return candidate
|
||||
return ""
|
||||
return (
|
||||
okf.nearest_listable_directory(bundle, path, dimension=dimension),
|
||||
okf.nearest_subdirectories(bundle, path, dimension=dimension),
|
||||
)
|
||||
|
||||
|
||||
def _refused_mapping(exc: Exception) -> dict[str, Any]:
|
||||
|
|
@ -1332,9 +1332,17 @@ def navigator_tools(
|
|||
# link -- still propagates untouched, because a refusal is a statement about the CALLER's
|
||||
# path and a failure to read something that IS there is not one.
|
||||
if not resolved.exists():
|
||||
ancestor, neighbours = _neighbours(bundle_dir, path, dimension)
|
||||
# P21/C2: the nearest listable ancestor AND up to five of its own subdirectories. The
|
||||
# ancestor alone says which rung to go back to; the neighbours say which names that
|
||||
# rung actually uses — measured, one run spent eleven calls walking ``R761/4-3``,
|
||||
# ``4.3``, ``4-2``, ``4-1``, ``4-0``, ``4-5``, ``4-6`` while the real names were
|
||||
# ``R761/4``, ``R761/41``, ``R761/42``. Omitted when the ancestor has no
|
||||
# subdirectories: an empty list would be a sentence with nothing in it.
|
||||
nearby = f" (its subdirectories include {', '.join(neighbours)})" if neighbours else ""
|
||||
raise okf.BundlePathNotFound(
|
||||
f"knowledge base {bundle_id!r} has no document {path!r}; nearest directory that "
|
||||
f"holds documents: {_nearest_listable(bundle_dir, path, dimension)!r} — list it "
|
||||
f"holds documents: {ancestor!r}{nearby} — list it "
|
||||
"with read_dir (it takes a filter) and read_file one of the names it gives"
|
||||
)
|
||||
# The verdict layer, refused HOWEVER the path was found (order 20260904T172353Z). No
|
||||
|
|
@ -1405,7 +1413,7 @@ def navigator_tools(
|
|||
assert opened is not None and requirements is not None # the constructor guard above
|
||||
# The base is resolved by the SAME index every read rung uses, so an unknown base is
|
||||
# refused here exactly as it is there rather than being accepted into the record.
|
||||
_resolve_bundle(index, bundle_id)
|
||||
resolved_dir = _resolve_bundle(index, bundle_id)
|
||||
read_paths = [call.path for call in opened if call.name == "read_file" and call.path]
|
||||
if path not in read_paths:
|
||||
raise RequirementNotRead(
|
||||
|
|
@ -1413,6 +1421,41 @@ def navigator_tools(
|
|||
"this is not one of them. Read it first — a requirement nobody read cannot bind a "
|
||||
"direction"
|
||||
)
|
||||
# P21/C1: the declaration must have LOOKED. MEASURED over round 4's six debate traces --
|
||||
# 13 declarations, and the distinct documents opened before each were
|
||||
# 1,1,1,1,1,1,1,2,5,5,6,13,13. Seven of them opened exactly ONE document, the base's front
|
||||
# matter, and declared its FIRST requirement (``Krav 1.2-1``, ``Krav 1.1-1``,
|
||||
# ``Krav 1.1.1-1``); none of the 13 named a fasit concept. Three of five paid runs read a
|
||||
# single document all run.
|
||||
#
|
||||
# THE ORDER OFFERED A SECOND RULE AND THE MEASUREMENT CHOSE BETWEEN THEM. The alternative
|
||||
# -- "the declared document must have been returned by a ``read_dir`` filtered on a word
|
||||
# from the approach's label" -- was replayed against the real listings and refuses 13 of
|
||||
# 13, including Soraasen's ``12.11``, which the order names as the closest any run came.
|
||||
# A gate that refuses every measured case, right and wrong alike, cannot discriminate: it
|
||||
# is the vacuous gate's mirror image. This one refuses 8 of 13 and keeps the five that
|
||||
# navigated, ``12.11`` among them.
|
||||
#
|
||||
# THE THRESHOLD IS NOT ON A CLIFF: k = 3, 4 and 5 all refuse the same eight, because the
|
||||
# measured distribution has a gap between 2 and 5. Three is the lowest of that plateau,
|
||||
# which is the least this can refuse while still separating the two measured classes.
|
||||
#
|
||||
# CAPPED BY THE BASE ITSELF. A base with two concept documents can be read whole in two,
|
||||
# and a floor above its size would make declaration impossible there -- a gate that can
|
||||
# only refuse. The cap is read off ``context_files``, the same property every listing rung
|
||||
# is built from, so the verdict layer is outside the denominator exactly as it is outside
|
||||
# the listings.
|
||||
distinct = len(set(read_paths))
|
||||
in_base = len(okf.navigate_bundle(resolved_dir).context_files)
|
||||
floor = min(_MIN_DOCUMENTS_READ, in_base)
|
||||
if distinct < floor:
|
||||
raise RequirementNotRead(
|
||||
f"{path!r}; this run has opened {distinct} distinct document(s) of the "
|
||||
f"{in_base} in {bundle_id!r}, and a binding requirement declared after {distinct} "
|
||||
f"is a guess rather than a finding — read at least {floor} before declaring one. "
|
||||
"Use read_dir with a 'filter' word from the approach's own label to find the "
|
||||
"candidates, then read_file the ones that could bind it"
|
||||
)
|
||||
requirements.append(DeclaredRequirement(bundle_id=bundle_id, path=path, ref=ref))
|
||||
# P20/A1: give back the DOCUMENT's own title and number, read off the base rather than
|
||||
# echoed from the arguments. MEASURED (P19 round 3, P17b): 13 declarations over 5 runs and
|
||||
|
|
|
|||
|
|
@ -1289,6 +1289,89 @@ def _matches_filter(file: BundleFile, needle: str) -> bool:
|
|||
return any(needle in value.casefold() for value in hay)
|
||||
|
||||
|
||||
#: P21/C2 — how many neighbouring directories a "no such path" refusal may name. A refusal's job is
|
||||
#: to hand back the one thing the caller can act on, not to re-list the level: five is enough to
|
||||
#: show the SHAPE of the names this base uses (``4``/``41``/``42`` rather than ``4-3``), and the
|
||||
#: full level is one ``read_dir`` away.
|
||||
_NEIGHBOUR_LIMIT: Final = 5
|
||||
|
||||
|
||||
def nearest_listable_directory(bundle: Bundle, path: str, *, dimension: str | None = None) -> str:
|
||||
"""The deepest ANCESTOR of ``path`` that ``directory_listing`` will actually answer for.
|
||||
|
||||
Chosen off the NAVIGATED ``context_files`` rather than off the filesystem, for two reasons that
|
||||
are the same reason: a directory can exist on disk and hold no navigated concept (nothing links
|
||||
it), in which case ``read_dir`` refuses it and the refusal would have handed the caller a path
|
||||
that does not resolve — ``_index_excerpt``'s rule one rung up, a path that never was is worse
|
||||
than no path. And ``context_files`` is the property that drops the ``type: verdict`` layer, so a
|
||||
refusal can never advertise by name the one layer no listing mentions.
|
||||
|
||||
Falls back to ``""``, the base's own top level, which ``directory_listing`` always answers.
|
||||
"""
|
||||
reachable = [f for f in bundle.context_files if in_dimension(f, dimension)]
|
||||
segments = path.strip("/").split("/")
|
||||
for depth in range(len(segments) - 1, 0, -1):
|
||||
candidate = "/".join(segments[:depth])
|
||||
if any(f.name.startswith(candidate + "/") for f in reachable):
|
||||
return candidate
|
||||
return ""
|
||||
|
||||
|
||||
def nearest_subdirectories(
|
||||
bundle: Bundle,
|
||||
path: str,
|
||||
*,
|
||||
dimension: str | None = None,
|
||||
limit: int = _NEIGHBOUR_LIMIT,
|
||||
) -> tuple[str, ...]:
|
||||
"""The directories a caller who named a path this base does not hold could have meant (P21/C2).
|
||||
|
||||
**The measured defect.** Over round 4's six traces, 16 of 105 ``read_dir`` calls and 2 of 38
|
||||
``read_file`` calls named a path the base does not hold, and eleven of those were one run
|
||||
walking ``R761/4-3``, ``R761/4.3``, ``R761/4-2``, ``R761/4-1``, ``R761/4-0``, ``R761/4-5``,
|
||||
``R761/4-6`` — guessing at a chapter-number spelling the corpus does not use, while the real
|
||||
neighbours are ``R761/4``, ``R761/41``, ``R761/42``. The refusal already named the nearest
|
||||
LISTABLE ancestor, which is the right rung; what it could not say is which of that rung's names
|
||||
the caller was reaching for.
|
||||
|
||||
**Every name it returns RESOLVES.** Built from ``context_files`` and through the SAME
|
||||
``in_dimension`` predicate the listing uses: a suggestion read off the filesystem could name a
|
||||
directory ``read_dir`` then refuses, and one built from ``files`` could name the ``type:
|
||||
verdict`` layer by path — advertising in a refusal the one layer no listing mentions.
|
||||
|
||||
**Ranked by longest common prefix with the segment that failed, then shortest, then name.** A
|
||||
ranking cannot refuse anything — this is help text on a refusal — so its failure direction is
|
||||
benign: at worst it names five real directories that are not the one meant. Prefix ranking is
|
||||
what puts ``4`` ahead of ``41`` for ``4-3``; with no common prefix at all every candidate ties
|
||||
and the order degrades to "the shortest names at this level", which is an honest "here is what
|
||||
IS here".
|
||||
"""
|
||||
ancestor = nearest_listable_directory(bundle, path, dimension=dimension)
|
||||
prefix = f"{ancestor}/" if ancestor else ""
|
||||
depth = len(prefix.split("/")) - 1 if prefix else 0
|
||||
segments = path.strip("/").split("/")
|
||||
missing = segments[depth] if len(segments) > depth else ""
|
||||
|
||||
children: set[str] = set()
|
||||
for file in bundle.context_files:
|
||||
if not in_dimension(file, dimension) or not file.name.startswith(prefix):
|
||||
continue
|
||||
rest = file.name[len(prefix) :].split("/")
|
||||
if len(rest) > 1:
|
||||
children.add(rest[0])
|
||||
|
||||
def _shared(name: str) -> int:
|
||||
n = 0
|
||||
for a, b in zip(name, missing):
|
||||
if a != b:
|
||||
break
|
||||
n += 1
|
||||
return n
|
||||
|
||||
ranked = sorted(children, key=lambda n: (-_shared(n), len(n), n))
|
||||
return tuple(f"{prefix}{name}" for name in ranked[:limit])
|
||||
|
||||
|
||||
def directory_listing(
|
||||
bundle: Bundle,
|
||||
path: str = "",
|
||||
|
|
@ -1404,9 +1487,16 @@ def directory_listing(
|
|||
f"{path!r} in knowledge base {bundle.dir!r} is a document, not a directory; "
|
||||
f"use read_file to read {named.name!r} whole"
|
||||
)
|
||||
# P21/C2: name the rung the caller could have meant. The SAME two helpers the read_file
|
||||
# refusal uses (kø-(p)) — one question about one base must not have two answers — and both
|
||||
# are built from ``context_files`` through ``in_dimension``, so every name handed back is
|
||||
# one this same function would then answer for.
|
||||
neighbours = nearest_subdirectories(bundle, path, dimension=dimension)
|
||||
ancestor = nearest_listable_directory(bundle, path, dimension=dimension)
|
||||
nearby = f"; its subdirectories include {', '.join(neighbours)}" if neighbours else ""
|
||||
raise BundlePathNotFound(
|
||||
f"knowledge base {bundle.dir!r} has no directory {path!r}; it holds no concept "
|
||||
"document under that path"
|
||||
f"document under that path. Nearest directory that does: {ancestor!r}{nearby}"
|
||||
)
|
||||
# A1: the WINDOW. Clamped, never refused — a caller asking for more than the maximum asked for
|
||||
# a listing, and the bound is this rung's job to keep, not the caller's to remember. A negative
|
||||
|
|
|
|||
|
|
@ -366,8 +366,11 @@ def test_the_requirement_gate_reads_the_second_bases_own_opened_list(tmp_path: P
|
|||
assert refusal["refusal"] == "RequirementNotRead", refusal
|
||||
assert not reqs_2, "base 2 recorded a requirement it never read"
|
||||
|
||||
tools_2["read_file"].func(bundle_2, doc_2)
|
||||
opened_2.append(explore.ToolCall(name="read_file", bundle_id=bundle_2, path=doc_2))
|
||||
# P21/C1: a declaration also needs the run to have LOOKED — at least three distinct documents
|
||||
# of the base, capped by its size — so the correction here is the same one a real run makes.
|
||||
for name in [f.name for f in okf.navigate_bundle(second).context_files][:3]:
|
||||
tools_2["read_file"].func(bundle_2, name)
|
||||
opened_2.append(explore.ToolCall(name="read_file", bundle_id=bundle_2, path=name))
|
||||
accepted = tools_2["declare_requirement"].func(bundle_2, doc_2, "Krav 1")
|
||||
assert accepted.get("declared") is True, accepted
|
||||
assert [r.path for r in reqs_2] == [doc_2]
|
||||
|
|
|
|||
|
|
@ -110,13 +110,24 @@ def test_a_requirement_the_run_never_opened_is_refused_and_recorded_nowhere() ->
|
|||
|
||||
|
||||
def test_the_correction_is_to_read_it_and_then_it_is_accepted() -> None:
|
||||
"""(b) The refusal is a turn the model can correct — the half that makes (a) a gate."""
|
||||
"""(b) The refusal is a turn the model can correct — the half that makes (a) a gate.
|
||||
|
||||
P21/C1 added a SECOND precondition — the run must have opened at least three distinct
|
||||
documents (capped by the base's own size) — so the correction is now "read it, and read enough
|
||||
of the base to have looked". The arm keeps its meaning: a declaration the run's own trace
|
||||
supports is ACCEPTED and RECORDED.
|
||||
"""
|
||||
tools, opened, declared = _wired()
|
||||
path = _a_concept()
|
||||
from portfolio_optimiser import okf
|
||||
|
||||
files = [f.name for f in okf.navigate_bundle(str(_TUNNEL)).context_files]
|
||||
path = files[0]
|
||||
body = tools["read_file"].func(bundle_id="tunnel-hauglia", path=path)
|
||||
assert not body.startswith("REFUSED"), body[:120]
|
||||
# The recorder is middleware in a real run; here the trace is appended directly, which is the
|
||||
# SAME list the tool reads.
|
||||
for other in files[1:3]:
|
||||
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=other))
|
||||
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
|
||||
answer = tools["declare_requirement"].func(
|
||||
bundle_id="tunnel-hauglia", path=path, ref="Krav 12.1"
|
||||
|
|
|
|||
282
tests/test_declaration_and_neighbours_loadbearing.py
Normal file
282
tests/test_declaration_and_neighbours_loadbearing.py
Normal file
|
|
@ -0,0 +1,282 @@
|
|||
"""P21 DEL C — a declaration that must have LOOKED, and a refusal that names the neighbours.
|
||||
|
||||
**C1, and the order asked for a measurement rather than a preference.** P19 DEL A made a direction
|
||||
name the requirement that binds it, and the rung works — round 4 produced 13 declarations over six
|
||||
runs. Not one named a fasit concept, and the distinct documents opened before each were
|
||||
``1,1,1,1,1,1,1,2,5,5,6,13,13``: seven declared the base's FIRST requirement after opening exactly
|
||||
ONE document.
|
||||
|
||||
The order offered two rules and asked which discriminates. Replayed against the real listings:
|
||||
|
||||
* **"the declared document must have been returned by a ``read_dir`` filtered on a word from the
|
||||
approach's label"** refuses **13 of 13** — including Søråsen's ``12.11``, which the order names
|
||||
as the closest any run came. Measured, ZERO of the 13 declarations were reached through a
|
||||
filtered listing at all. A gate that refuses every measured case, right and wrong alike, cannot
|
||||
discriminate; it is the vacuous gate's mirror image.
|
||||
* **"fewer than k distinct documents opened"** at k=3 refuses **8 of 13** and keeps the five that
|
||||
navigated, ``12.11`` among them. k=3, 4 and 5 refuse the SAME eight — the distribution has a gap
|
||||
between 2 and 5 — so the threshold is not on a cliff, and 3 is the lowest of that plateau.
|
||||
|
||||
The second is built. It is capped by the base's own document count, so a two-document base stays
|
||||
declarable rather than becoming a level nobody can declare on.
|
||||
|
||||
**C2.** Over the same six traces, 18 of 143 path-bearing tool calls named a path the base does not
|
||||
hold, and ELEVEN of them are one run walking ``R761/4-3``, ``4.3``, ``4-2``, ``4-1``, ``4-0``,
|
||||
``4-5``, ``4-6`` — guessing a chapter-number spelling the corpus does not use, while the real names
|
||||
are ``R761/4``, ``R761/41``, ``R761/42``. The refusal already named the nearest LISTABLE ancestor,
|
||||
which is the right rung; what it could not say is which of that rung's names was meant. Measured
|
||||
after: **16 of 18** refusals now name at least one neighbour, and the two that do not are paths
|
||||
whose ancestor holds documents and no subdirectories — an empty list is omitted rather than
|
||||
rendered as a sentence with nothing in it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser import okf
|
||||
from portfolio_optimiser.explore import ToolCall, navigator_tools
|
||||
|
||||
_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
|
||||
_TUNNEL = _EXAMPLES / "tunnel-hauglia"
|
||||
|
||||
|
||||
def _wired(bundle_dir: Path = _TUNNEL) -> tuple[dict[str, Any], list[ToolCall], list[Any]]:
|
||||
opened: list[ToolCall] = []
|
||||
declared: list[Any] = []
|
||||
tools = navigator_tools((str(bundle_dir),), opened=opened, requirements=declared)
|
||||
return {t.name: t for t in tools}, opened, declared
|
||||
|
||||
|
||||
def _concepts(bundle_dir: Path = _TUNNEL) -> list[str]:
|
||||
return [f.name for f in okf.navigate_bundle(str(bundle_dir)).context_files]
|
||||
|
||||
|
||||
# --- C1 -------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_c1_a_declaration_after_one_document_is_refused_with_the_denominator() -> None:
|
||||
"""The KNOWN-POSITIVE: exactly the measured failure — open one, declare it.
|
||||
|
||||
Both numbers are in the refusal, because the pair is the diagnosis: "1 of 5" says something
|
||||
"1" alone does not, and a refusal that stated neither would leave the model guessing how much
|
||||
more is wanted (``BudgetExceeded``'s kø-(y) rule, one rung down).
|
||||
"""
|
||||
tools, opened, declared = _wired()
|
||||
path = _concepts()[0]
|
||||
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
|
||||
|
||||
answer = tools["declare_requirement"].func(
|
||||
bundle_id="tunnel-hauglia", path=path, ref="Krav 1.1—1"
|
||||
)
|
||||
|
||||
assert answer["refusal"] == "RequirementNotRead"
|
||||
assert "1 distinct document(s)" in answer["refused"]
|
||||
assert "of the 5" in answer["refused"], answer["refused"]
|
||||
assert "at least 3" in answer["refused"]
|
||||
assert declared == [], "a refused declaration must leave no record"
|
||||
|
||||
|
||||
def test_c1_the_same_declaration_after_three_documents_is_accepted() -> None:
|
||||
"""The KNOWN-NEGATIVE, and the half that keeps C1 from being a gate that only refuses.
|
||||
|
||||
Same base, same declared document, same ``ref`` — the ONLY difference is how much of the base
|
||||
the run had opened. Without this arm the refusal above would be indistinguishable from one that
|
||||
fires on everything, which is exactly what the rule the measurement rejected does.
|
||||
"""
|
||||
tools, opened, declared = _wired()
|
||||
files = _concepts()
|
||||
for name in files[:3]:
|
||||
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=name))
|
||||
|
||||
answer = tools["declare_requirement"].func(
|
||||
bundle_id="tunnel-hauglia", path=files[0], ref="Krav 1.1—1"
|
||||
)
|
||||
|
||||
assert answer["declared"] is True
|
||||
assert len(declared) == 1
|
||||
|
||||
|
||||
def test_c1_the_same_path_read_three_times_is_still_one_document() -> None:
|
||||
"""DISTINCT paths, not calls. Re-reading one document is not navigation, and a rule counting
|
||||
calls would be satisfied by the cheapest possible loop."""
|
||||
tools, opened, declared = _wired()
|
||||
path = _concepts()[0]
|
||||
for _ in range(3):
|
||||
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
|
||||
|
||||
answer = tools["declare_requirement"].func(
|
||||
bundle_id="tunnel-hauglia", path=path, ref="Krav 1.1—1"
|
||||
)
|
||||
|
||||
assert answer["refusal"] == "RequirementNotRead"
|
||||
assert "1 distinct document(s)" in answer["refused"]
|
||||
assert declared == []
|
||||
|
||||
|
||||
def test_c1_a_small_base_stays_declarable(tmp_path: Path) -> None:
|
||||
"""Capped by the base's OWN size: two documents can be read whole in two.
|
||||
|
||||
A fixed floor above a base's size would make declaration impossible there — a gate that can
|
||||
only refuse, on exactly the small fixtures this repo is built on.
|
||||
"""
|
||||
base = tmp_path / "mini"
|
||||
base.mkdir()
|
||||
(base / "index.md").write_text(
|
||||
"---\nbundle_id: mini\n---\n\n- [A](a.md) — one.\n- [B](b.md) — two.\n", encoding="utf-8"
|
||||
)
|
||||
(base / "a.md").write_text("---\ntype: Krav\ntitle: A\n---\n\nbody A\n", encoding="utf-8")
|
||||
(base / "b.md").write_text("---\ntype: Krav\ntitle: B\n---\n\nbody B\n", encoding="utf-8")
|
||||
|
||||
tools, opened, declared = _wired(base)
|
||||
for name in ("a.md", "b.md"):
|
||||
opened.append(ToolCall(name="read_file", bundle_id="mini", path=name))
|
||||
|
||||
answer = tools["declare_requirement"].func(bundle_id="mini", path="a.md", ref="1")
|
||||
|
||||
assert answer["declared"] is True
|
||||
assert len(declared) == 1
|
||||
|
||||
|
||||
def test_c1_the_never_opened_refusal_still_fires_first() -> None:
|
||||
"""The P19 gate is UNCHANGED and still the first thing checked: "you never read this" and "you
|
||||
have barely read anything" are different facts, and the first is the one the caller can fix
|
||||
with one call."""
|
||||
tools, opened, declared = _wired()
|
||||
files = _concepts()
|
||||
for name in files[:3]:
|
||||
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=name))
|
||||
|
||||
answer = tools["declare_requirement"].func(
|
||||
bundle_id="tunnel-hauglia", path=files[4], ref="Krav 1.1—1"
|
||||
)
|
||||
|
||||
assert answer["refusal"] == "RequirementNotRead"
|
||||
assert "is not one of them" in answer["refused"]
|
||||
assert declared == []
|
||||
|
||||
|
||||
# --- C2 -------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _r761_shaped(root: Path) -> Path:
|
||||
"""A base shaped like the delivered R761: numbered chapter directories under one top level.
|
||||
|
||||
Crafted rather than mounted, for MAJOR-3's reason — R761 lives outside this repository and
|
||||
cannot be a test dependency — and shaped from the MEASURED names (``R761/4``, ``R761/41``,
|
||||
``R761/42``) so the arm tests the ranking the real corpus provoked.
|
||||
"""
|
||||
base = root / "proc"
|
||||
(base / "R761" / "4").mkdir(parents=True)
|
||||
(base / "R761" / "41").mkdir()
|
||||
(base / "R761" / "42").mkdir()
|
||||
links = "".join(f"- [{d}](R761/{d}/k.md) — chapter {d}.\n" for d in ("4", "41", "42"))
|
||||
(base / "index.md").write_text(f"---\nbundle_id: proc\n---\n\n{links}", encoding="utf-8")
|
||||
for d in ("4", "41", "42"):
|
||||
(base / "R761" / d / "k.md").write_text(
|
||||
f"---\ntype: Krav\ntitle: Chapter {d}\nprosessnr: '{d}'\n---\n\nbody {d}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return base
|
||||
|
||||
|
||||
def test_c2_a_guessed_chapter_spelling_is_answered_with_the_real_names(tmp_path: Path) -> None:
|
||||
"""The order's known-positive: ``R761/4-3`` must come back with ``R761/4…`` candidates.
|
||||
|
||||
``R761/4`` is FIRST, because the ranking is longest-common-prefix with the segment that failed
|
||||
— which is the one thing that distinguishes "the names at this level" from "the name you were
|
||||
reaching for".
|
||||
"""
|
||||
base = _r761_shaped(tmp_path)
|
||||
bundle = okf.navigate_bundle(str(base))
|
||||
|
||||
neighbours = okf.nearest_subdirectories(bundle, "R761/4-3")
|
||||
|
||||
assert neighbours[0] == "R761/4"
|
||||
assert set(neighbours) == {"R761/4", "R761/41", "R761/42"}
|
||||
|
||||
|
||||
def test_c2_the_read_file_refusal_carries_them(tmp_path: Path) -> None:
|
||||
"""The seam: the message the MODEL sees. A helper nothing calls would leave every arm above
|
||||
green and the measured defect untouched."""
|
||||
base = _r761_shaped(tmp_path)
|
||||
tools, _opened, _declared = _wired(base)
|
||||
|
||||
answer = tools["read_file"].func(bundle_id="proc", path="R761/4-3/k.md")
|
||||
|
||||
assert answer.startswith("REFUSED (BundlePathNotFound)")
|
||||
assert "'R761'" in answer
|
||||
assert "subdirectories include R761/4, R761/41, R761/42" in answer
|
||||
|
||||
|
||||
def test_c2_the_read_dir_refusal_carries_them_too(tmp_path: Path) -> None:
|
||||
"""The OTHER rung, and the reason both were changed: a caller that guesses a directory gets the
|
||||
same help as one that guesses a document. One question about one base, one answer."""
|
||||
base = _r761_shaped(tmp_path)
|
||||
tools, _opened, _declared = _wired(base)
|
||||
|
||||
answer = tools["read_dir"].func(bundle_id="proc", path="R761/4-3")
|
||||
|
||||
assert answer["refusal"] == "BundlePathNotFound"
|
||||
assert "R761/4, R761/41, R761/42" in answer["refused"]
|
||||
|
||||
|
||||
def test_c2_an_ancestor_with_no_subdirectories_names_none(tmp_path: Path) -> None:
|
||||
"""Omission, never an empty clause. MEASURED: 2 of the 18 guessed paths land here — their
|
||||
ancestor holds documents and no directories — and a trailing "its subdirectories include "
|
||||
with nothing after it is a sentence that says nothing."""
|
||||
base = _r761_shaped(tmp_path)
|
||||
tools, _opened, _declared = _wired(base)
|
||||
|
||||
answer = tools["read_file"].func(bundle_id="proc", path="R761/4/missing.md")
|
||||
|
||||
assert answer.startswith("REFUSED (BundlePathNotFound)")
|
||||
assert "subdirectories include" not in answer
|
||||
|
||||
|
||||
def test_c2_every_name_it_hands_back_resolves(tmp_path: Path) -> None:
|
||||
"""The property that makes the help worth having (``_index_excerpt``'s rule): a path that never
|
||||
was is worse than no path. Each suggestion is fed straight back to ``read_dir``."""
|
||||
base = _r761_shaped(tmp_path)
|
||||
tools, _opened, _declared = _wired(base)
|
||||
bundle = okf.navigate_bundle(str(base))
|
||||
|
||||
for name in okf.nearest_subdirectories(bundle, "R761/4-3"):
|
||||
listing = tools["read_dir"].func(bundle_id="proc", path=name)
|
||||
assert "refused" not in listing, f"{name} does not resolve: {listing}"
|
||||
|
||||
|
||||
def test_c2_the_verdict_layer_is_never_advertised_by_name(tmp_path: Path) -> None:
|
||||
"""Built from ``context_files``, never ``files``: the one layer no listing mentions must not
|
||||
appear in a REFUSAL either, which would be the same leak wearing an apology."""
|
||||
base = tmp_path / "withverdict"
|
||||
(base / "judged").mkdir(parents=True)
|
||||
(base / "open").mkdir()
|
||||
(base / "index.md").write_text(
|
||||
"---\nbundle_id: wv\n---\n\n- [A](open/a.md) — one.\n- [V](judged/v.md) — two.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(base / "open" / "a.md").write_text(
|
||||
"---\ntype: Krav\ntitle: A\n---\n\nbody\n", encoding="utf-8"
|
||||
)
|
||||
(base / "judged" / "v.md").write_text(
|
||||
"---\ntype: verdict\ntitle: V\n---\n\nprior judgement\n", encoding="utf-8"
|
||||
)
|
||||
bundle = okf.navigate_bundle(str(base))
|
||||
|
||||
neighbours = okf.nearest_subdirectories(bundle, "nope")
|
||||
|
||||
assert neighbours == ("open",), neighbours
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", ["", "a"])
|
||||
def test_c2_a_top_level_guess_names_the_top_level_directories(tmp_path: Path, path: str) -> None:
|
||||
"""The fallback: no ancestor at all means the base's own top level, which ``directory_listing``
|
||||
always answers. ``krav`` and ``r761`` were both measured as top-level guesses on R761."""
|
||||
base = _r761_shaped(tmp_path)
|
||||
bundle = okf.navigate_bundle(str(base))
|
||||
|
||||
assert okf.nearest_subdirectories(bundle, path) == ("R761",)
|
||||
|
|
@ -38,6 +38,7 @@ What each arm pins:
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
|
@ -60,8 +61,22 @@ def _wired(bundle_dir: Path) -> tuple[dict[str, Any], list[ToolCall], list[Decla
|
|||
return {t.name: t for t in tools}, opened, declared
|
||||
|
||||
|
||||
def _declare(tools: dict[str, Any], opened: list[ToolCall], base: str, path: str) -> dict[str, Any]:
|
||||
"""Read it the way a run does, then declare it — the (b) path of P19 DEL A."""
|
||||
def _declare(
|
||||
tools: dict[str, Any],
|
||||
opened: list[ToolCall],
|
||||
base: str,
|
||||
path: str,
|
||||
*,
|
||||
also: Sequence[str] = (),
|
||||
) -> dict[str, Any]:
|
||||
"""Read it the way a run does, then declare it — the (b) path of P19 DEL A.
|
||||
|
||||
``also`` carries the OTHER documents the run opened. P21/C1 made "it opened enough of the base
|
||||
to have looked" part of the declaration's precondition, capped by the base's own size, so a
|
||||
crafted two-document base needs no extras while a five-document example does.
|
||||
"""
|
||||
for other in also:
|
||||
opened.append(ToolCall(name="read_file", bundle_id=base, path=other))
|
||||
opened.append(ToolCall(name="read_file", bundle_id=base, path=path))
|
||||
return tools["declare_requirement"].func(bundle_id=base, path=path, ref="Krav 4.1.2-1")
|
||||
|
||||
|
|
@ -130,8 +145,9 @@ def test_a_verdict_document_is_never_named_back_by_title(tmp_path: Path) -> None
|
|||
def test_the_tunnel_example_answers_its_own_title_and_no_number() -> None:
|
||||
"""(a) control on a REAL example base: title present, reference number honestly absent."""
|
||||
tools, opened, _declared = _wired(_TUNNEL)
|
||||
path = okf.navigate_bundle(str(_TUNNEL)).context_files[0].name
|
||||
answer = _declare(tools, opened, "tunnel-hauglia", path)
|
||||
files = [f.name for f in okf.navigate_bundle(str(_TUNNEL)).context_files]
|
||||
path = files[0]
|
||||
answer = _declare(tools, opened, "tunnel-hauglia", path, also=files[1:3])
|
||||
assert answer["title"].startswith("Kilder: tunnelbelysning")
|
||||
assert answer["req_number"] == "", "this base declares none, and the reply must not invent one"
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue