open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(inbox): walk the drop directory recursively

Browse source 
Door B listed `inbox.iterdir()` and kept only top-level files. A file in a
subdirectory was neither ingested nor refused: it appeared in none of the
result's buckets, so a nested drop produced a bundle that was silently short
of what was dropped and no count said so. That broke the K1b identity for any
inbox with folders in it. Operator decision 2026-09-06.

- `walk_inbox` is the ONE walk rule, shared with `tools/okf_corpus_run.py`:
  the denominator N is now counted over exactly the set of files the door
  ingests, rather than over a second listing that happened to agree.
- Sorted on the whole relative path, not the basename, so the order is a
  function of the tree; that is what keeps rebuild-from-scratch byte-equal to
  an incremental update.
- A concept's `source_file` is the path relative to the inbox root,
  `/`-separated. The concept NAME still comes from the basename, so two
  folders holding one basename hit the existing §3 collision refusal instead
  of one silently claiming the other's concept.
- Dot-directories and a bundle directory inside the inbox are skipped with a
  CODE, in a new `InboxResult.skipped`. Recursion makes the door's own output
  reachable as its own input; a silent skip would be the same
  absence-without-a-denominator defect one level down.
- `--path-prefix` reduces per component and rejoins with `/`, so the caller
  driving a nested corpus can carry the relative directory. Reducing the whole
  string folded the separator into a `-` and flattened `sub/sub2`.

`tests/test_inbox_flow.py::test_subdirectories_are_not_walked` asserted the
opposite and is superseded in place, with the reason written down.

Measured on the K2 corpus (flat, N=43): 39/43 merged, 4 coded, K1b holds. The
bundle digest is
`1472e98aec8643c5beee540f4c42b5e437bd26e7c61d69a91bcff799f06a6d13` over 1108
files -- byte-identical to a run of the same corpus at 190086f WITHOUT this
change (`diff -r` exit 0), so recursion costs a flat inbox nothing. It differs
from the stored 2026-09-03 artifact by one line in `index.md`
(`- [Corpus run history](log.md)`), which 95eb271 added 15 hours after that
bundle was built.

Suite 1113 passed, `ruff` clean, `mypy --strict src/ tools/` clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 04:11:00 +02:00
commit aa87eb8818
11 changed files with 489 additions and 39 deletions
Download patch file Download diff file Expand all files Collapse all files

9
CLAUDE.md
View file

@ -13,7 +13,14 @@ one boundary rule:
needs go via commons, never edited locally. The library ships the §11
golden fixtures (byte-exact) for the three door-A source types
(`ingest-golden-{file,sql,http}/`, shipped in `9dd86b1`).
- **Door B — bundle inbox:** converts dropped files to OKF concepts. All
- **Door B — bundle inbox:** converts dropped files to OKF concepts. The drop
directory is walked RECURSIVELY, sorted by relative path, and a concept's
`source_file` is that relative path (`/`-separated) while its NAME still
comes from the basename — so a nested duplicate hits the §3 collision
refusal rather than vanishing. Dot-directories and a bundle nested inside
the inbox are skipped with a code, never silently, because recursion makes
the door's own output reachable as its own input (operator 2026-09-06; the
flat listing was not a boundary, it was an absence with no denominator). All
file-type→text extraction lives HERE (the guard is text-only). v1 core:
`md`, `txt`, `csv`, `json`, `html` (stdlib). `pdf`/`docx`/`xlsx` only via
the optional `[extract]` extra; without it those types are rejected

4
README.md
View file

@ -68,6 +68,10 @@ bundle:
`pdf` and the five office formats (`docx`, `xlsx`, `pptx`, `odt`, `rtf`)
require the optional `[extract]` extra and are rejected fail-fast without
it. Extracted text passes the security gate before anything is persisted.
The drop directory is walked **recursively**, in sorted relative-path order:
a file at any depth is ingested and records its path relative to the inbox
root as its `source_file`, while dot-directories and a bundle directory
sitting inside the inbox are skipped with a reported code.
<!-- extract-formats: .md, .txt, .csv, .json, .html, .htm, .pdf, .docx, .xlsx, .pptx, .odt, .rtf -->
3. **External bundle import.** Import and merge of third-party OKF bundles:

2
src/llm_ingestion_okf/__init__.py
View file

@ -52,6 +52,7 @@ from .inbox import (
GateDecision,
InboxResult,
PersistedFile,
SkippedPath,
process_inbox,
)
from .importer import (
@ -101,6 +102,7 @@ __all__ = [
"MergedConcept",
"NetworkGateError",
"PersistedFile",
"SkippedPath",
"RefusedConcept",
"RenderError",
"SourceError",

156
src/llm_ingestion_okf/inbox.py
View file

@ -20,6 +20,7 @@ supplies the verdict and this module only obeys it.
from __future__ import annotations
import hashlib
import os
import unicodedata
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass, replace
@ -275,6 +276,26 @@ class FailedFile:
error: IngestError
#: Why the walk refused to descend into a directory. A CODE rather than a log
#: line: a skipped directory is an outcome the caller has to be able to count,
#: and "we found nothing under here" is a measurement, not a fact about the
#: inbox.
SKIPPED_DOT_DIRECTORY = "skipped_dot_directory"
SKIPPED_BUNDLE_DIRECTORY = "skipped_bundle_directory"
@dataclass(frozen=True)
class SkippedPath:
"""A directory the recursive walk did not descend into, and why.
`path` is relative to the inbox root, `/`-separated, exactly like a
concept's `source_file`.
"""
path: str
code: str
@dataclass(frozen=True)
class InboxResult:
"""Every dropped file's outcome, in sorted filename order.
@ -294,6 +315,72 @@ class InboxResult:
# consumer's number means. Without segmentation the two are equal, which is
# what makes this additive rather than a second thing to keep in step.
concepts: tuple[PersistedFile, ...] = ()
# Directories the walk refused to enter, each with its code. Additive and
# last, so every existing positional construction and every existing
# consumer's four buckets keep their meaning: a skipped directory holds no
# dropped FILE outcome, it explains a set of files that were never dropped.
skipped: tuple[SkippedPath, ...] = ()
def relative_source(path: Path, inbox: Path) -> str:
"""A dropped file's name as the provenance layer records it.
Relative to the inbox root and `/`-separated, so the recorded name is the
same on every platform and two documents sharing a basename in different
folders stay distinguishable. For a flat inbox this is the bare filename,
which is why every existing bundle's bytes are unchanged.
"""
return path.relative_to(inbox).as_posix()
def walk_inbox(
inbox: Path, *, exclude: Path | None = None
) -> tuple[tuple[Path, ...], tuple[SkippedPath, ...]]:
"""Every dropped file at any depth, sorted by relative path, plus the skips.
ONE implementation, shared with `tools/okf_corpus_run.py`: the corpus
measurement counts the denominator N, and a walk that disagreed with the
door's would report a count over a different set of files than the one that
was ingested.
Sorted on the whole relative path rather than the basename -- the order has
to be a function of the TREE, or two inboxes holding the same documents
would walk them differently and their indexes would diverge.
Two directories are refused, both with a code. A dot-directory is
machinery, not documents. `exclude` is the bundle: with a flat listing a
bundle nested inside the inbox was invisible by accident, and recursion
removes the accident -- without the skip the door would extract its own
concepts and ingest them as documents on the next run.
"""
excluded = exclude.resolve() if exclude is not None else None
dropped: list[Path] = []
skipped: list[SkippedPath] = []
for parent_name, dirnames, filenames in os.walk(inbox):
parent = Path(parent_name)
kept: list[str] = []
for name in sorted(dirnames):
child = parent / name
if name.startswith("."):
skipped.append(
SkippedPath(path=relative_source(child, inbox), code=SKIPPED_DOT_DIRECTORY)
)
elif excluded is not None and child.resolve() == excluded:
skipped.append(
SkippedPath(path=relative_source(child, inbox), code=SKIPPED_BUNDLE_DIRECTORY)
)
else:
kept.append(name)
# In place: this is how `os.walk` is told not to descend, and a skipped
# directory must not be entered at all rather than entered and filtered.
dirnames[:] = kept
for name in filenames:
child = parent / name
if child.is_file():
dropped.append(child)
dropped.sort(key=lambda item: relative_source(item, inbox))
skipped.sort(key=lambda item: item.path)
return (tuple(dropped), tuple(skipped))
def _is_inbox_owned(path: Path) -> bool:
@ -454,6 +541,7 @@ def _render_segments(
gate: Gate,
profile: BundleProfile,
bundle_id: str,
source_file: str,
) -> BlockedFile | None:
"""Render every segment, or refuse the WHOLE document.
@ -467,7 +555,7 @@ def _render_segments(
A refusal is therefore reported once, for the document, rather than once
per segment: the operator's unit of review is the document they dropped.
"""
extractor_id = Path(path.name).suffix.lower().lstrip(".") or "none"
extractor_id = path.suffix.lower().lstrip(".") or "none"
assert_plan_applies(
plan,
source_sha256=hashlib.sha256(source_bytes).hexdigest(),
@ -492,7 +580,7 @@ def _render_segments(
]
if refused:
return BlockedFile(
source_file=path.name,
source_file=source_file,
disposition=refused[0].disposition,
reasons=tuple(reason for decision in refused for reason in decision.reasons),
)
@ -500,7 +588,7 @@ def _render_segments(
for entry, decision in decisions:
structure: DocumentStructure | None = None
if profile.index.facets is not None:
structure = derive_document_structure(decision.sanitized_text, source_file=path.name)
structure = derive_document_structure(decision.sanitized_text, source_file=source_file)
_validate_facets(structure, profile)
outputs.append(
(
@ -512,7 +600,7 @@ def _render_segments(
# segment's own first line: the plan is the record of the
# judgement, and a heading inside a slice is not it.
title=entry.title,
source_file=path.name,
source_file=source_file,
source_bytes=source_bytes,
ingested_at=entry.ingested_at,
profile=profile,
@ -623,10 +711,16 @@ def process_inbox(
inbox = Path(inbox_dir)
if not inbox.is_dir():
raise SourceError(f"inbox directory does not exist: {inbox}", code="source_root_missing")
bundle = Path(bundle_dir)
# Top-level only, sorted: the operator's nested structure is theirs, and a
# deterministic order is what makes a re-run comparable.
dropped = sorted((path for path in inbox.iterdir() if path.is_file()), key=lambda p: p.name)
# RECURSIVE, sorted by relative path. The operator's nested structure is
# theirs to arrange, but it is not theirs to lose: a file under a folder
# used to be neither ingested nor refused, so a bundle built from a nested
# drop was silently short of what was dropped.
dropped, skipped = walk_inbox(inbox, exclude=bundle)
def source_name(path: Path) -> str:
return relative_source(path, inbox)
persisted: list[PersistedFile] = []
concepts: list[PersistedFile] = []
@ -660,9 +754,10 @@ def process_inbox(
except OSError as exc:
failed.append(
FailedFile(
source_file=path.name,
source_file=source_name(path),
error=SourceError(
f"cannot read dropped file {path.name}: {exc}", code="source_file_missing"
f"cannot read dropped file {source_name(path)}: {exc}",
code="source_file_missing",
),
)
)
@ -673,11 +768,16 @@ def process_inbox(
matched_hashes.add(covering.source_sha256)
targets: tuple[str, ...]
if covering is None:
# From the BASENAME, not the relative path: the concept name
# is the bundle's, and folding a folder into it would rename
# every concept the moment an operator tidied their inbox. Two
# folders holding the same basename therefore collide, and the
# §3 gate below refuses both rather than picking a winner.
targets = (inbox_filename(inbox_slug(path.name), profile=profile),)
else:
targets = tuple(_check_segment_path(item.path) for item in covering.entries)
except IngestError as exc:
failed.append(FailedFile(source_file=path.name, error=exc))
failed.append(FailedFile(source_file=source_name(path), error=exc))
continue
named.append((path, targets, source_bytes, covering is not None))
for target in targets:
@ -707,17 +807,17 @@ def process_inbox(
# five colliding paths is one thing the operator has to fix, and five
# identical entries would report the same rename five times.
for path in sorted(
{owner for name in contested for owner in slug_owners[name]}, key=lambda item: item.name
{owner for name in contested for owner in slug_owners[name]}, key=source_name
):
claimed = sorted(name for name in contested if path in slug_owners[name])
others = sorted(
{other.name for name in claimed for other in slug_owners[name] if other != path}
{source_name(other) for name in claimed for other in slug_owners[name] if other != path}
)
failed.append(
FailedFile(
source_file=path.name,
source_file=source_name(path),
error=MaterializationError(
f"{path.name!r} and {', '.join(repr(other) for other in others)}"
f"{source_name(path)!r} and {', '.join(repr(other) for other in others)}"
f" both reduce to {', '.join(repr(name) for name in claimed)}"
" — rename one; refusing to pick a winner",
code="inbox_slug_collision",
@ -728,7 +828,6 @@ def process_inbox(
# Phase 2: the §3 ownership scan, evaluated against the bundle as it was
# BEFORE this run — a file written below must never be mistaken for
# pre-existing curated content by a later file's check.
bundle = Path(bundle_dir)
owned_by_source: dict[str, set[str]] = {}
if profile.segmentation is not None:
# RECURSIVE, and only here. A nested concept is invisible to a flat
@ -763,7 +862,7 @@ def process_inbox(
if unstamped:
failed.append(
FailedFile(
source_file=path.name,
source_file=source_name(path),
error=MaterializationError(
f"generated filename {unstamped[0]!r} collides with an existing file "
"that does not carry the inbox marker — refusing to overwrite curated "
@ -776,7 +875,9 @@ def process_inbox(
outputs: list[tuple[str, str, tuple[str, ...]]] = []
try:
text = extract_text(
path.name, source_bytes, renderer=_resolve_renderer(profile, path.name)
source_name(path),
source_bytes,
renderer=_resolve_renderer(profile, path.name),
)
covering = _plan_covering(plans, source_bytes)
if covering is not None:
@ -789,6 +890,7 @@ def process_inbox(
gate=gate,
profile=profile,
bundle_id=(root_frontmatter_values or {})[_bundle_id_key(profile)],
source_file=source_name(path),
)
if blocked is not None:
if blocked.disposition == _DISPOSITION_QUARANTINE:
@ -800,7 +902,7 @@ def process_inbox(
decision = gate(text)
if decision.disposition != _DISPOSITION_PERSIST:
blocked = BlockedFile(
source_file=path.name,
source_file=source_name(path),
disposition=decision.disposition,
reasons=decision.reasons,
)
@ -819,7 +921,7 @@ def process_inbox(
# deriving from bytes the gate rejected would put unscreened
# content in the frontmatter and the index.
structure = derive_document_structure(
decision.sanitized_text, source_file=path.name
decision.sanitized_text, source_file=source_name(path)
)
title = structure.title
_validate_facets(structure, profile)
@ -830,7 +932,7 @@ def process_inbox(
decision.sanitized_text,
okf_type=okf_type,
title=title,
source_file=path.name,
source_file=source_name(path),
source_bytes=source_bytes,
ingested_at=ingested_at,
profile=profile,
@ -842,15 +944,16 @@ def process_inbox(
except OSError as exc:
failed.append(
FailedFile(
source_file=path.name,
source_file=source_name(path),
error=SourceError(
f"cannot read dropped file {path.name}: {exc}", code="source_file_missing"
f"cannot read dropped file {source_name(path)}: {exc}",
code="source_file_missing",
),
)
)
continue
except IngestError as exc:
failed.append(FailedFile(source_file=path.name, error=exc))
failed.append(FailedFile(source_file=source_name(path), error=exc))
continue
bundle.mkdir(parents=True, exist_ok=True)
@ -859,7 +962,9 @@ def process_inbox(
# creates one. Without this the very first hierarchical write fails.
(bundle / target_name).parent.mkdir(parents=True, exist_ok=True)
written = write_bytes(bundle, target_name, content)
concepts.append(PersistedFile(source_file=path.name, path=written, reasons=reasons))
concepts.append(
PersistedFile(source_file=source_name(path), path=written, reasons=reasons)
)
# One entry per SOURCE FILE, whatever the document expanded into. That
# is what `persisted` has always meant, so an existing consumer's count
# does not change under a profile that segments.
@ -870,7 +975,7 @@ def process_inbox(
# document owned before and does not now is retired HERE, after the
# writes, so a failure above leaves the previous round intact.
_retire_stale_segments(
bundle, owned_by_source.get(path.name, set()) - set(targets), profile
bundle, owned_by_source.get(source_name(path), set()) - set(targets), profile
)
# §6 index — the last disk mutation, and only when something was written.
@ -907,6 +1012,7 @@ def process_inbox(
rejected=tuple(rejected),
failed=tuple(sorted(failed, key=lambda entry: entry.source_file)),
concepts=tuple(concepts),
skipped=skipped,
)

37
tests/test_corpus_run.py
View file

@ -52,6 +52,43 @@ def test_the_denominator_is_the_directory_not_a_literal(tmp_path: Path) -> None:
assert report.n == len(list(root.iterdir()))
def test_the_denominator_counts_files_in_subdirectories_too(tmp_path: Path) -> None:
"""N is the corpus, and the corpus is the whole tree.
The measurement and the door must walk by the SAME rule -- `walk_inbox` is
the one implementation -- or the report would state a denominator over a
different set of files than the one that was ingested, and the
conservation identity would hold over the wrong N.
"""
root = corpus(tmp_path, {"a.md": SUBSTANTIVE})
(root / "sub").mkdir()
(root / "sub" / "b.md").write_text(SUBSTANTIVE, encoding="utf-8", newline="")
report = okf_corpus_run.measure(root, tmp_path / "bundle", ingested_at=INGESTED_AT)
assert report.n == 2
assert report.persisted == 2
# `unaccounted` is `dropped - merged - coded`: it can only be empty if the
# measurement's names and the door's names are the SAME strings, which is
# what pins the two walks to one rule rather than to two that agree today.
assert report.unaccounted == ()
# Re-extracted through `corpus / source_file`, so the relative name has to
# resolve back to the file it came from.
assert report.substantive == 2
def test_the_denominator_skips_a_bundle_written_inside_the_corpus(tmp_path: Path) -> None:
root = corpus(tmp_path, {"a.md": SUBSTANTIVE})
bundle = root / "bundle"
okf_corpus_run.measure(root, bundle, ingested_at=INGESTED_AT)
second = okf_corpus_run.measure(root, bundle, ingested_at=INGESTED_AT)
assert second.n == 1
assert second.persisted == 1
assert second.unaccounted == ()
def test_the_conservation_identity_holds_and_the_run_exits_zero(tmp_path: Path) -> None:
root = corpus(
tmp_path,

27
tests/test_docs_promises.py
View file

@ -82,3 +82,30 @@ def test_the_readme_still_states_what_stays_out() -> None:
text = README.read_text(encoding="utf-8")
assert ".doc`" in text or "Word 97" in text
assert ".doc" not in set(_PANDOC_FORMATS)
def test_the_readme_recursion_claim_matches_the_door() -> None:
"""The README says the drop directory is walked recursively. A sentence is
not a mechanism, so both halves are asserted here: the claim is in the
prose, and the door actually does it. Either one alone can go stale --
prose that outlived the code is the failure this whole module exists for.
"""
import tempfile
text = README.read_text(encoding="utf-8")
assert "walked **recursively**" in text
from llm_ingestion_okf.inbox import GateDecision, process_inbox
with tempfile.TemporaryDirectory() as workspace:
inbox = Path(workspace) / "inbox" / "sub"
inbox.mkdir(parents=True)
(inbox / "deep.md").write_text("Body\n", encoding="utf-8")
result = process_inbox(
Path(workspace) / "inbox",
Path(workspace) / "bundle",
"2026-09-07T08:00:00Z",
okf_type="reference",
gate=lambda body: GateDecision(sanitized_text=body, disposition="warn", reasons=()),
)
assert [item.source_file for item in result.persisted] == ["sub/deep.md"]

17
tests/test_inbox_flow.py
View file

@ -421,14 +421,21 @@ def test_an_empty_inbox_writes_nothing(tmp_path: Path) -> None:
assert not bundle.exists() or list(bundle.iterdir()) == []
def test_subdirectories_are_not_walked(tmp_path: Path) -> None:
"""Top-level only, like every other inbox in this ecosystem: a nested tree
is the operator's structure, not ours to flatten into one namespace.
def test_subdirectories_are_walked(tmp_path: Path) -> None:
"""Superseded, deliberately, by the operator's decision of 2026-09-06.
This test used to assert the opposite -- top-level only, "the operator's
nested structure is theirs". The rule it pinned was not a boundary, it was
a silent loss: a file under a folder appeared in NONE of the result's
buckets, so the count said nothing was there rather than that nothing was
looked at. The nested structure is still the operator's, and it survives
in the recorded `source_file`; it is the disappearance that is gone. The
rule's own suite is `tests/test_inbox_recursion.py`.
"""
drop(tmp_path / "inbox" / "nested", "deep.md", "Body\n")
drop(tmp_path / "inbox", "top.md", "Body\n")
result, bundle = run(tmp_path, StubGuard())
assert [entry.source_file for entry in result.persisted] == ["top.md"]
assert not (bundle / "inbox-deep.md").exists()
assert [entry.source_file for entry in result.persisted] == ["nested/deep.md", "top.md"]
assert (bundle / "inbox-deep.md").is_file()

205
tests/test_inbox_recursion.py Normal file
View file

@ -0,0 +1,205 @@
"""Door B walks the inbox RECURSIVELY (order 20260906T220349Z-6526414037).
Until this suite the door listed `inbox.iterdir()` and kept only top-level
files. A file in a subdirectory was neither ingested nor refused: it did not
appear in any of the result's buckets, so a bundle built from a nested drop
was silently short and no count said so. That is the absence-without-a-
denominator failure, and it broke the K1b identity (bundle == what was
dropped) for every operator whose inbox has folders in it.
What is pinned here:
- a file at any depth is ingested, and its `source_file` is the path RELATIVE
to the inbox root, `/`-separated, so two documents that share a basename in
different folders are still distinguishable in the provenance layer;
- the walk order is the sorted relative path, which is what keeps a
rebuild-from-scratch byte-identical to an incremental update (the K6 gate);
- a dot-directory and a bundle directory sitting INSIDE the inbox are skipped
with a CODE, never silently — the door's own output must not be re-ingested
as input, and an operator has to be able to see that it was not;
- the generated concept name still comes from the BASENAME, so two nested
files reducing to one name hit the existing §3 collision refusal rather
than one silently claiming the other's concept.
"""
from __future__ import annotations
import hashlib
from pathlib import Path
from llm_ingestion_okf.inbox import (
SKIPPED_BUNDLE_DIRECTORY,
SKIPPED_DOT_DIRECTORY,
GateDecision,
process_inbox,
)
from llm_ingestion_okf.profiles import STRUCTURED_V1
INGESTED_AT = "2026-09-07T08:00:00Z"
def _gate(text: str) -> GateDecision:
return GateDecision(sanitized_text=text, disposition="warn", reasons=())
def _run(inbox: Path, bundle: Path, **kwargs: object) -> object:
return process_inbox(
inbox,
bundle,
INGESTED_AT,
okf_type="reference",
gate=_gate,
**kwargs, # type: ignore[arg-type]
)
def _drop(inbox: Path, relative: str, body: str) -> None:
path = inbox / relative
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(body, encoding="utf-8")
def test_a_file_in_a_subdirectory_is_ingested_and_carries_its_relative_path(
tmp_path: Path,
) -> None:
inbox = tmp_path / "inbox"
inbox.mkdir()
_drop(inbox, "top.md", "Top level.\n")
_drop(inbox, "sub/one.md", "One deep.\n")
_drop(inbox, "sub/sub2/two.md", "Two deep.\n")
result = _run(inbox, tmp_path / "bundle")
assert [item.source_file for item in result.persisted] == [ # type: ignore[attr-defined]
"sub/one.md",
"sub/sub2/two.md",
"top.md",
]
assert result.failed == () # type: ignore[attr-defined]
assert result.rejected == () # type: ignore[attr-defined]
concept = (tmp_path / "bundle" / "inbox-two.md").read_text(encoding="utf-8")
assert "source_file: sub/sub2/two.md" in concept
def test_the_walk_order_is_the_sorted_relative_path(tmp_path: Path) -> None:
"""Sorted on the WHOLE relative path, not the basename.
Sorting on the basename would order `b/a.md` before `a/z.md`, which makes
the order depend on names rather than on the tree — two inboxes holding
the same files would walk them differently and the index would differ.
"""
inbox = tmp_path / "inbox"
inbox.mkdir()
_drop(inbox, "b/aaa.md", "B then aaa.\n")
_drop(inbox, "a/zzz.md", "A then zzz.\n")
result = _run(inbox, tmp_path / "bundle")
assert [item.source_file for item in result.persisted] == [ # type: ignore[attr-defined]
"a/zzz.md",
"b/aaa.md",
]
def test_a_dot_directory_is_skipped_with_a_code(tmp_path: Path) -> None:
inbox = tmp_path / "inbox"
inbox.mkdir()
_drop(inbox, "kept.md", "Kept.\n")
_drop(inbox, ".git/config.md", "Not a document.\n")
_drop(inbox, ".git/objects/deep.md", "Not a document either.\n")
result = _run(inbox, tmp_path / "bundle")
assert [item.source_file for item in result.persisted] == ["kept.md"] # type: ignore[attr-defined]
assert [(item.path, item.code) for item in result.skipped] == [ # type: ignore[attr-defined]
(".git", SKIPPED_DOT_DIRECTORY)
]
def test_a_bundle_directory_inside_the_inbox_is_skipped_with_a_code(tmp_path: Path) -> None:
"""The door's own output is not its own input.
With a flat listing a nested bundle was invisible by accident. Recursion
removes that accident: without this skip the second run would extract the
concepts written by the first, ingest them as documents, and grow the
bundle on every run.
"""
inbox = tmp_path / "inbox"
inbox.mkdir()
bundle = inbox / "bundle"
_drop(inbox, "doc.md", "A document.\n")
first = _run(inbox, bundle)
assert [item.source_file for item in first.persisted] == ["doc.md"] # type: ignore[attr-defined]
assert first.skipped == () # type: ignore[attr-defined]
second = _run(inbox, bundle)
assert [item.source_file for item in second.persisted] == ["doc.md"] # type: ignore[attr-defined]
assert [(item.path, item.code) for item in second.skipped] == [ # type: ignore[attr-defined]
("bundle", SKIPPED_BUNDLE_DIRECTORY)
]
def test_two_nested_files_sharing_a_basename_are_refused_not_silently_merged(
tmp_path: Path,
) -> None:
"""The concept name still comes from the basename, so this is a collision.
Stated as a test rather than left implicit: recursion makes it reachable
for the first time, and the answer is the §3 refusal the door already
gives — never a winner picked by walk order.
"""
inbox = tmp_path / "inbox"
inbox.mkdir()
_drop(inbox, "a/same.md", "First.\n")
_drop(inbox, "b/same.md", "Second.\n")
result = _run(inbox, tmp_path / "bundle")
assert result.persisted == () # type: ignore[attr-defined]
assert [item.source_file for item in result.failed] == ["a/same.md", "b/same.md"] # type: ignore[attr-defined]
assert {item.error.code for item in result.failed} == {"inbox_slug_collision"} # type: ignore[attr-defined]
def _tree_hash(root: Path) -> str:
digest = hashlib.sha256()
for path in sorted(root.rglob("*")):
if not path.is_file():
continue
digest.update(path.relative_to(root).as_posix().encode("utf-8"))
digest.update(b"\0")
digest.update(path.read_bytes())
digest.update(b"\0")
return digest.hexdigest()
def test_a_nested_rebuild_from_scratch_equals_the_incremental_bundle(tmp_path: Path) -> None:
"""The K6 identity, over an inbox with subdirectories.
The faceted index is a projection of the WHOLE bundle, so it is exactly
where a walk order that depended on arrival would show up. Building one
bundle in two rounds and another in one must produce the same bytes.
"""
inbox = tmp_path / "inbox"
inbox.mkdir()
_drop(inbox, "sub/one.md", "# Ett\n\nFirst document.\n")
_drop(inbox, "sub/sub2/two.md", "# To\n\nSecond document.\n")
incremental = tmp_path / "incremental"
_run(inbox, incremental, profile=STRUCTURED_V1)
_drop(inbox, "three.md", "# Tre\n\nThird document.\n")
_run(inbox, incremental, profile=STRUCTURED_V1)
scratch = tmp_path / "scratch"
_run(inbox, scratch, profile=STRUCTURED_V1)
# The control FIRST: two empty trees hash the same, so an equality that
# ran before recursion existed would have been a green over nothing.
assert sorted(path.name for path in scratch.rglob("*.md")) == [
"inbox-one.md",
"inbox-three.md",
"inbox-two.md",
"index.md",
]
assert _tree_hash(incremental) == _tree_hash(scratch)

35
tests/test_propose_segments.py
View file

@ -304,6 +304,41 @@ def test_a_path_prefix_scopes_every_entry_under_one_directory(tmp_path: Path) ->
assert parse_segmentation_plan(payload).entries
def test_a_multi_component_prefix_carries_the_relative_directory(tmp_path: Path) -> None:
"""Door B now walks the inbox recursively and records a `source_file`
relative to its root, so the caller driving a nested corpus has a
DIRECTORY, not a name, to scope by. Each component is reduced on its own
and rejoined with `/`: reducing the whole string would fold the separator
into a `-` and flatten `sub/sub2` into one component named `sub-sub2`,
which is a different bundle shape than the inbox it came from.
"""
out = tmp_path / "plan.json"
assert (
okf_propose_segments.main(
[str(write(tmp_path)), "--out", str(out), "--path-prefix", "Bilag 3/Del II"]
)
== 0
)
payload = json.loads(out.read_text(encoding="utf-8"))
assert payload["entries"]
for entry in payload["entries"]:
assert entry["path"].startswith("bilag-3/del-ii/")
assert parse_segmentation_plan(payload).entries
def test_a_prefix_with_one_empty_component_is_refused(tmp_path: Path) -> None:
"""`a//b` and `a/###/b` are the multi-component form of the same defect the
single-component gate already refuses: an empty component would collapse
the path silently rather than scope it.
"""
for prefix in ("sub//two", "sub/###/two", "/sub", "sub/"):
code = okf_propose_segments.main(
[str(write(tmp_path)), "--out", str(tmp_path / "p.json"), "--path-prefix", prefix]
)
assert code == 2, prefix
assert not (tmp_path / "p.json").exists(), prefix
def test_without_a_prefix_the_artifact_is_byte_identical(tmp_path: Path) -> None:
"""Additive. Every plan already produced stays exactly what it was."""
first = tmp_path / "a.json"

14
tools/okf_corpus_run.py
View file

@ -44,7 +44,13 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from llm_ingestion_okf.errors import IngestError # noqa: E402
from llm_ingestion_okf.extract import extract_text # noqa: E402
from llm_ingestion_okf.inbox import GateDecision, InboxResult, process_inbox # noqa: E402
from llm_ingestion_okf.inbox import ( # noqa: E402
GateDecision,
InboxResult,
process_inbox,
relative_source,
walk_inbox,
)
from llm_ingestion_okf.profiles import ( # noqa: E402
SEGMENTED_OKF_V0_2,
STRUCTURED_V1,
@ -289,7 +295,11 @@ def measure(
Keyword-only with defaults, so the flat call that produced the published
K1/K2 numbers stays source-compatible and byte-identical.
"""
dropped = tuple(sorted(path.name for path in corpus.iterdir() if path.is_file()))
# ONE walk rule, imported rather than restated: the denominator has to be
# counted over exactly the set of files the door ingests, or the
# conservation identity would hold over a different N than the run did.
walked, _ = walk_inbox(corpus, exclude=bundle)
dropped = tuple(relative_source(path, corpus) for path in walked)
started = time.monotonic()
result = process_inbox(
corpus,

22
tools/okf_propose_segments.py
View file

@ -612,13 +612,22 @@ def run(
# Reduced HERE, before anything is read: a prefix that survives to the
# entries as an empty component would produce exactly the unscoped paths
# the caller asked to avoid, and would do it silently.
scope = reduce_to_id_grammar(path_prefix) if path_prefix else ""
if path_prefix and not scope:
#
# PER COMPONENT, because the prefix carries a DIRECTORY now that Door B
# walks the inbox recursively and records a relative `source_file`.
# Reducing the whole string would fold `/` into a `-` and flatten
# `sub/sub2` into the single component `sub-sub2` -- a bundle shaped unlike
# the inbox it came from, and unlike what the caller wrote.
components = (
[reduce_to_id_grammar(part) for part in path_prefix.split("/")] if path_prefix else []
)
if path_prefix and not all(components):
raise ProposerError(
f"--path-prefix {path_prefix!r} reduces to nothing under the id grammar "
"([a-z0-9][a-z0-9-]*); refusing to write unscoped paths under a scope "
"that was asked for"
f"--path-prefix {path_prefix!r} has a component that reduces to nothing under "
"the id grammar ([a-z0-9][a-z0-9-]*); refusing to write unscoped paths under a "
"scope that was asked for"
)
scope = "/".join(components)
if not source.is_file():
raise ProposerError(f"source is not a file: {source}")
try:
@ -677,7 +686,8 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"--path-prefix",
default="",
help=(
"scope every entry's path under this directory. Required for a corpus: "
"scope every entry's path under this directory, `/`-separated for a "
"nested one (each component is reduced on its own). Required for a corpus: "
"section numbering is document-local, so two documents propose the same "
"path and Door B refuses both. An argument rather than something this "
"tool derives -- it sees one document and cannot know what else is in "