portfolio-optimiser/tests/test_read_file_directory_refusal_loadbearing.py
Kjell Tore Guttormsen eb4137415c feat(prepass,explore): padding dies at the prompt; a refusal the model can act on is a return value [skip-docs]
Order 20260908T195801Z. Findings 4 and 5 from the S7 acid test, then the two things
finding 99 measured and deliberately did not fix (D3, D2).

No user-facing surface changes: no new flag, no new command, no changed output
contract. Both seams are internal (the pre-pass rendering, and the shape a tool
answers a model with), so [skip-docs] rather than a README edit that would describe
nothing an operator can do differently.

FINDING 4 -- MEASURED, NOTHING BUILT. K2's price schedule IS readable without
guessing (8 column spans, 71 of 91 non-blank rows give >= 2 cells, the split stable
for K = 2..64). But 0 of 92 rows name all three of code/quantity/unit_cost -- also
under a looser substring match -- and 0 of 91 data rows carry code + quantity +
amount. The triple is not formatted away; it is not in the document. It is a price
SUMMARY plus nine rate cards whose unit-price columns are empty (pre-award). The
order's binding decision rule therefore falls against building:
--derive-cost-baseline keeps refusing, and MAJOR-4's own honesty limit holds.

FINDING 5 -- BUILT. Measured on the actual rendering path (concept_text, not the
raw file): the delivered excerpt is 104 lines / 67 245 chars, carrying 208 interior
whitespace runs, 117 of them >= 100 and the longest 887 -- 56 806 of 67 245
characters = 84.5 %, over 72 of 104 lines. collapse_padding, called from
_data_blocks (the one renderer both arms share, and therefore AFTER
verify_against_bundle -- collapsing in concept_text would break every payload's own
digest), gives -72.4 %: line count invariant, non-whitespace byte-identical, leading
indentation untouched, no number changed.

F99-D3 -- read_file / read_dir / read_bundle now RETURN their refusal. MAF turns a
tool raise into "Error: Function failed." (_tools.py:1410-1432, :1427) and counts it
against DEFAULT_MAX_CONSECUTIVE_ERRORS_PER_REQUEST = 3, so everything the refusing
arm knows is destroyed on the way out. The gates are unchanged; the property they
exist for -- the reason travels, the bytes never do -- is now asserted explicitly on
the returned value. The arm is keyed on named classes, never bare Exception, because
ExplorationError is itself a RuntimeError subclass.

F99-D2 -- the invariant row, plus one for finding 5 (a stated deviation from "one
row only": finding 5 is a separately built seam and the ledger's standing rule
requires its own row).

19 existing arms rewritten, never deleted and never weakened: where the class
carried a distinction, the refusal KIND carries it now.

13 mutations, all red against the whole suite (W1-W5, M1-M8), each restored from
scratchpad with shasum -c. Control 1543 passed / 5 skipped (from 1529/5, a strict
superset, 0 removed). Golden demo-transcript.stdout unchanged
(shasum -a 1 of the CONTENT = ea8c534773acdbe41ae68f2c55724d69aaf8be4f).

Measurement: docs/2026-09-08-funn-4-5-og-read-nekt.md

Co-Authored-By: Claude <Opus 5>
2026-09-08 23:36:54 +02:00

147 lines
7.8 KiB
Python

"""``read_file`` on a DIRECTORY is refused by name and pointed at ``read_dir``.
Finding (c) of the live K2 session (``docs/2026-09-06-major2-levende-k2.md`` § 3/§ 4): a live model
walking the ladder ``list_bundles -> read_bundle -> read_dir -> read_file`` reached a level holding
subdirectories (``30-1``, ``30-7``, ``521-001`` ...) and then called ``read_file`` on one of them.
Three such calls in one run.
**What was CHECKED FIRST, and what it found.** The order allows for two causes and asks which one
holds. It is the second. A listing DOES already separate the two kinds structurally: every
``directory_listing`` payload answers with ``directories`` (entries keyed ``path``, carrying a
subtree ``documents`` count) and ``documents`` (entries keyed ``name``, carrying ``type``, ``title``
and ``chars``) as two distinct keys, and no directory ever appears among the documents. Arm (4)
pins that, because it is a property this file now depends on rather than one it introduces. So no
trailing-``/`` marker is added: the shape already says which is which, and changing the payload
would move a listing three older gates measure byte for byte.
What was missing is the OTHER half. MEASURED before this work, on the shipped nested example base:
read_file(id, "a") -> builtins.IsADirectoryError: [Errno 21] Is a directory: '<abs path>'
An ``OSError``, so it lands on the crash channel rather than the CLI's refusal tuple and hosting's
400 arm, and it says nothing about which rung the caller should have used. The refusal now names
both -- that the path is a directory, and ``read_dir`` -- following ``VerdictLayerRefused`` and
``DimensionScopeRefused``: the rule lives in the TOOL, so it holds for both callers (the
exploration and, since S2c, the debate), and it is a ``ValueError`` because the caller is a model
choosing a path, not a program that is broken.
**MEASURED, REPORTED, NOT FIXED (outside this order).** The live call was
``read_file(".../30-7.md")`` -- the model appended ``.md`` to a directory NAME, which resolves to a
path that does not exist at all, not to the directory. Measured on the same base:
read_file(id, "a.md") -> builtins.FileNotFoundError: [Errno 2] No such file or directory
That is a second, adjacent defect with the same shape (an OSError on the crash channel where a
named refusal belongs), and the fix ordered here does not cover it. Arm (5) pins the measurement so
the gap is a fact in the suite rather than a sentence in a report.
Detach point: remove the directory branch from ``read_file`` -> RED on (1), (2) and (3).
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
import pytest
from portfolio_optimiser.explore import DirectoryPathRefused, navigator_tools
_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
#: The only NESTED base shipped -- the one that HAS a directory to ask for.
_NESTED = _EXAMPLES / "nav-golden-hierarchy" / "bundle"
def _tools(bundle_dir: Path) -> dict[str, Any]:
return {t.name: t for t in navigator_tools((str(bundle_dir),))}
async def _invoke(tool: Any, **arguments: Any) -> str:
"""A tool's answer as text (S2c's helper: ``invoke`` returns ``[Content]``, and a test that
stringified the list would compare object reprs and pass against anything)."""
return "".join(getattr(c, "text", "") or "" for c in await tool.invoke(arguments=arguments))
def _a_directory(bundle_dir: Path) -> str:
"""A directory path taken out of the listing itself -- the same way a model gets one."""
listing = _tools(bundle_dir)["read_bundle"].func(bundle_id=bundle_dir.name)
assert listing["directories"], "fixture has no subdirectory to ask for"
return str(listing["directories"][0]["path"])
def test_a_directory_path_is_refused_and_the_other_rung_is_named() -> None:
"""(1) the refusal says WHAT the path is and WHICH tool reads it."""
path = _a_directory(_NESTED)
message = _tools(_NESTED)["read_file"].func(bundle_id=_NESTED.name, path=path)
assert message.startswith(f"REFUSED ({DirectoryPathRefused.__name__})"), (
"the refusal is not distinguishable from a document's body"
)
assert path in message
assert "directory" in message
assert "read_dir" in message, "a refusal that does not name the other rung strands the caller"
def test_the_refusal_is_a_value_error_not_an_os_error() -> None:
"""(2) the channel, which is the half a message alone cannot carry.
``ValueError`` is the ``BundlePathNotFound``/``DimensionScopeRefused`` precedent -- the CLI's
refusal tuple and hosting's 400 arm. The ``OSError`` assert is what tells a REPLACEMENT from a
wrapper: re-raising ``IsADirectoryError`` with a better message would pass arm (1).
"""
path = _a_directory(_NESTED)
answer = _tools(_NESTED)["read_file"].func(bundle_id=_NESTED.name, path=path)
# F99-D3: the refusal is RETURNED now, so the branch is identified by the kind it names. The
# channel assertion moves onto the class itself and keeps both halves: it is on the CLI's
# refusal tuple, and it is NOT a re-raised ``IsADirectoryError`` wearing a better message --
# an ``OSError`` is outside ``_RETURNABLE_REFUSALS`` and would propagate rather than answer.
assert answer.startswith(f"REFUSED ({DirectoryPathRefused.__name__})")
assert issubclass(DirectoryPathRefused, ValueError)
assert not issubclass(DirectoryPathRefused, OSError)
async def test_the_refusal_holds_through_the_real_tool_surface() -> None:
"""(3) reached through ``invoke``, not only through ``.func``.
The seam a model actually calls is the tool, and a check that only ran on the plain function
would be absent from the one caller it was written for.
"""
path = _a_directory(_NESTED)
answer = await _invoke(_tools(_NESTED)["read_file"], bundle_id=_NESTED.name, path=path)
assert answer.startswith(f"REFUSED ({DirectoryPathRefused.__name__})")
assert "read_dir" in answer
def test_a_listing_already_separates_directories_from_documents() -> None:
"""(4) the "check first" arm: the payload's shape is what distinguishes the two kinds.
Also the anti-vacuity control for this file -- it proves the fixture really does hold both
kinds, so arm (1) is asking for a directory rather than for nothing.
"""
listing = _tools(_NESTED)["read_bundle"].func(bundle_id=_NESTED.name)
directories = {d["path"] for d in listing["directories"]}
documents = {d["name"] for d in listing["documents"]}
assert directories and documents, "fixture must hold both kinds for this to mean anything"
assert directories.isdisjoint(documents)
assert all(set(d) == {"path", "documents"} for d in listing["directories"])
assert all(set(d) == {"name", "type", "title", "chars"} for d in listing["documents"])
def test_a_document_is_still_returned_whole() -> None:
"""CONTROL: the branch is a gate, not a wall -- the rung it guards still works."""
listing = _tools(_NESTED)["read_bundle"].func(bundle_id=_NESTED.name)
name = listing["documents"][0]["name"]
body = _tools(_NESTED)["read_file"].func(bundle_id=_NESTED.name, path=name)
assert body.strip(), "reading a real document must be untouched by the directory branch"
def test_a_nonexistent_sibling_is_still_an_os_error() -> None:
"""(5) MEASURED, REPORTED, NOT FIXED: the live call's ACTUAL shape.
The model appended ``.md`` to a directory name, which is not a directory -- it is a path that
does not exist. This order fixes the directory case; this arm records that the adjacent case is
unchanged, so the gap cannot be mistaken for covered. Written as an assertion on the CURRENT
behaviour: when it goes red, someone has closed it, and that is a decision to be recorded.
"""
path = _a_directory(_NESTED) + ".md"
with pytest.raises(FileNotFoundError):
_tools(_NESTED)["read_file"].func(bundle_id=_NESTED.name, path=path)