feat(inbox): walk the drop directory recursively

Door B listed `inbox.iterdir()` and kept only top-level files. A file in a
subdirectory was neither ingested nor refused: it appeared in none of the
result's buckets, so a nested drop produced a bundle that was silently short
of what was dropped and no count said so. That broke the K1b identity for any
inbox with folders in it. Operator decision 2026-09-06.

- `walk_inbox` is the ONE walk rule, shared with `tools/okf_corpus_run.py`:
  the denominator N is now counted over exactly the set of files the door
  ingests, rather than over a second listing that happened to agree.
- Sorted on the whole relative path, not the basename, so the order is a
  function of the tree; that is what keeps rebuild-from-scratch byte-equal to
  an incremental update.
- A concept's `source_file` is the path relative to the inbox root,
  `/`-separated. The concept NAME still comes from the basename, so two
  folders holding one basename hit the existing §3 collision refusal instead
  of one silently claiming the other's concept.
- Dot-directories and a bundle directory inside the inbox are skipped with a
  CODE, in a new `InboxResult.skipped`. Recursion makes the door's own output
  reachable as its own input; a silent skip would be the same
  absence-without-a-denominator defect one level down.
- `--path-prefix` reduces per component and rejoins with `/`, so the caller
  driving a nested corpus can carry the relative directory. Reducing the whole
  string folded the separator into a `-` and flattened `sub/sub2`.

`tests/test_inbox_flow.py::test_subdirectories_are_not_walked` asserted the
opposite and is superseded in place, with the reason written down.

Measured on the K2 corpus (flat, N=43): 39/43 merged, 4 coded, K1b holds. The
bundle digest is
`1472e98aec8643c5beee540f4c42b5e437bd26e7c61d69a91bcff799f06a6d13` over 1108
files -- byte-identical to a run of the same corpus at 190086f WITHOUT this
change (`diff -r` exit 0), so recursion costs a flat inbox nothing. It differs
from the stored 2026-09-03 artifact by one line in `index.md`
(`- [Corpus run history](log.md)`), which 95eb271 added 15 hours after that
bundle was built.

Suite 1113 passed, `ruff` clean, `mypy --strict src/ tools/` clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 04:11:00 +02:00
commit aa87eb8818
11 changed files with 489 additions and 39 deletions

View file

@ -44,7 +44,13 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from llm_ingestion_okf.errors import IngestError # noqa: E402
from llm_ingestion_okf.extract import extract_text # noqa: E402
from llm_ingestion_okf.inbox import GateDecision, InboxResult, process_inbox # noqa: E402
from llm_ingestion_okf.inbox import ( # noqa: E402
GateDecision,
InboxResult,
process_inbox,
relative_source,
walk_inbox,
)
from llm_ingestion_okf.profiles import ( # noqa: E402
SEGMENTED_OKF_V0_2,
STRUCTURED_V1,
@ -289,7 +295,11 @@ def measure(
Keyword-only with defaults, so the flat call that produced the published
K1/K2 numbers stays source-compatible and byte-identical.
"""
dropped = tuple(sorted(path.name for path in corpus.iterdir() if path.is_file()))
# ONE walk rule, imported rather than restated: the denominator has to be
# counted over exactly the set of files the door ingests, or the
# conservation identity would hold over a different N than the run did.
walked, _ = walk_inbox(corpus, exclude=bundle)
dropped = tuple(relative_source(path, corpus) for path in walked)
started = time.monotonic()
result = process_inbox(
corpus,

View file

@ -612,13 +612,22 @@ def run(
# Reduced HERE, before anything is read: a prefix that survives to the
# entries as an empty component would produce exactly the unscoped paths
# the caller asked to avoid, and would do it silently.
scope = reduce_to_id_grammar(path_prefix) if path_prefix else ""
if path_prefix and not scope:
#
# PER COMPONENT, because the prefix carries a DIRECTORY now that Door B
# walks the inbox recursively and records a relative `source_file`.
# Reducing the whole string would fold `/` into a `-` and flatten
# `sub/sub2` into the single component `sub-sub2` -- a bundle shaped unlike
# the inbox it came from, and unlike what the caller wrote.
components = (
[reduce_to_id_grammar(part) for part in path_prefix.split("/")] if path_prefix else []
)
if path_prefix and not all(components):
raise ProposerError(
f"--path-prefix {path_prefix!r} reduces to nothing under the id grammar "
"([a-z0-9][a-z0-9-]*); refusing to write unscoped paths under a scope "
"that was asked for"
f"--path-prefix {path_prefix!r} has a component that reduces to nothing under "
"the id grammar ([a-z0-9][a-z0-9-]*); refusing to write unscoped paths under a "
"scope that was asked for"
)
scope = "/".join(components)
if not source.is_file():
raise ProposerError(f"source is not a file: {source}")
try:
@ -677,7 +686,8 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"--path-prefix",
default="",
help=(
"scope every entry's path under this directory. Required for a corpus: "
"scope every entry's path under this directory, `/`-separated for a "
"nested one (each component is reduced on its own). Required for a corpus: "
"section numbering is document-local, so two documents propose the same "
"path and Door B refuses both. An argument rather than something this "
"tool derives -- it sees one document and cannot know what else is in "