open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(corpus): the root index links the bundle's own log

Browse source 
Measured on the K2 artifact by a consumer: `log.md` was on disk and no
index named it, so a reader entering the bundle at `index.md` -- the walk
section 8 exists to support -- never reached the one file carrying `N`.

Stated as a LOCAL choice rather than conformance, because it is one.
Upstream's own bundles do not link their log: measured at `9a15b13`, 0 of
the 24 shipped `index.md` files name the single `log.md` in the set, with
the same grep form finding `tables/index.md` in 4 of them as the
known-positive control. That shows the link is not REQUIRED -- not that
it is disallowed. `docs/plan/okf-v0.2-alignment.md` P1-F6 already
recorded the upstream shape; a line there now separates the two claims,
since reserved names still stay out of an `entries_match_directory`
listing and this profile has that off.

It lives in the harness because the library cannot make it. The log's
content IS the run's outcome, so it cannot exist when the indexes are
projected, and an index that enumerated it off the directory would gain
the link only from the second run onward -- breaking
rebuild-equals-incremental, the property the segmented bundle is built
on.

The membership test is load-bearing and was measured, not assumed. The
two reprojections disagree about this line: the per-directory one drops
every managed entry before re-emitting its block, while the flat one
keeps a managed line whose target is not an owned concept, deliberately,
so that a regex cannot delete curated content. Appending unconditionally
therefore doubled the entry on the second unsegmented run, which is why
both run modes are pinned separately.

1052 -> 1054 tests. `mypy --strict` clean, `ruff` clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-03 20:23:56 +02:00
commit 95eb271f57
4 changed files with 180 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

14
CHANGELOG.md
View file

@ -17,6 +17,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
dated from `ingested_at`, so `merged + sum(coded rejections) == N` is
checkable from the bundle alone rather than only from a report that does not
travel with it. Without `--plans-dir` a run is unchanged.
- **The corpus harness links that `log.md` from the bundle's root `index.md`.**
Measured on the K2 artifact by a consumer: the log was on disk and no index
named it, so a reader entering the bundle at `index.md` -- the walk section 8
exists to support -- never reached the one file carrying `N`. A LOCAL choice
rather than conformance, and stated as one: upstream's own bundles do not
link their log (measured at `9a15b13`, 0 of 24 shipped `index.md` files name
the single `log.md` in the set), which shows the link is not required, not
that it is disallowed. It lives in the harness because the library cannot
make it -- the log's content is the run's outcome, so an index that
enumerated it off the directory would gain the link only from the second run
and break rebuild-equals-incremental. Written only when absent, because the
two reprojections differ: the per-directory one drops the line as a managed
entry, the flat one keeps it (its target is not an owned concept), and
appending unconditionally doubled it on the second unsegmented run.
Measured cause: the harness passed `STRUCTURED_V1` and no plans, so a
43-document corpus arrived as 39 flat concepts with no `adjudication` key

10
docs/plan/okf-v0.2-alignment.md
View file

@ -619,6 +619,16 @@ profile requires has to know to exclude reserved names, and this library tells
them only about `index.md`. Consistent with "the library enumerates nothing" —
but the caller is being asked to know something we hold and do not state.
**This is not in tension with the corpus harness linking its own `log.md`
(`tools/okf_corpus_run.py`, 2026-09-03), and the two statements are about
different things.** Here: a reserved name must stay OUT of the listing an
`entries_match_directory` profile matches against, or the directory comparison
fails. There: a harness adds one navigational link to the root index of the
bundle it produces, under a profile with `entries_match_directory=False`. What
upstream's `acme_retail` settles is that the link is not required; it does not
make the link wrong, and a consumer entering a bundle at `index.md` otherwise
never reaches the file carrying `N`.
**What the sweep confirmed, which matters as much as what it found:**
- **V-A3 holds against real third-party v0.2 data.** No `generated` value in

105
tests/test_corpus_run.py
View file

@ -367,3 +367,108 @@ def test_a_second_run_into_the_same_bundle_reproduces_it_byte_for_byte(tmp_path:
if path.is_file()
}
assert second == first
def test_the_root_index_links_the_bundles_own_log(tmp_path: Path) -> None:
"""A log nothing links is a file on disk, not a member of the bundle.
Measured on the artifact: the K2 bundle carried a conformant root `log.md`
that no index named, so a consumer walking the bundle from `index.md` --
which is the walk section 8 exists to support -- never reached the one file
carrying `N`.
This is a LOCAL choice, not a conformance requirement, and the distinction
is worth keeping straight. Section 9 lets `log.md` sit at any level and
section 8 has an index enumerate its directory's contents, but upstream's
own reference bundles do not link it: measured at `9a15b13`, 0 of the 24
shipped `index.md` files name the single `log.md` in the bundle set. So
upstream proves the link is not required, not that it is disallowed.
It is made HERE, in the harness, because the library cannot make it. The
log's content is the run's outcome, so it cannot be written before the
indexes are projected -- and an index that enumerated `log.md` off the
directory would gain the link only on the SECOND run, breaking the
rebuild-equals-incremental property the segmented bundle is built on. The
harness instead writes the link after the log, and only when it is not
already there -- a test, not an append, because the two reprojections
disagree about this line: the per-directory one drops it as a managed
entry, the flat one keeps it because its target is not an owned concept.
"""
root = corpus(tmp_path, {"doc.md": SEGMENTABLE, "flat.md": SUBSTANTIVE})
plans = tmp_path / "plans"
plans.mkdir()
_propose(root / "doc.md", plans / "doc.json")
bundle = tmp_path / "bundle"
argv = [
"--corpus",
str(root),
"--report",
str(tmp_path / "r.md"),
"--bundle",
str(bundle),
"--ingested-at",
INGESTED_AT,
"--plans-dir",
str(plans),
"--bundle-id",
"k2",
"--okf-version",
"0.2",
]
assert okf_corpus_run.main(argv) == 0
index = (bundle / "index.md").read_text(encoding="utf-8")
assert "](log.md)" in index
assert index.count("](log.md)") == 1
# The link is to the log in THIS directory, so a nested index must not
# carry one: there is no `log.md` beside it to reach.
for nested in bundle.rglob("*/index.md"):
assert "](log.md)" not in nested.read_text(encoding="utf-8")
# Rebuild equals incremental, still. This is what discriminates the two
# ways the append could be wrong: a link the reprojection keeps would be
# doubled here, and one the harness forgot to re-write would vanish.
first = {
path.relative_to(bundle).as_posix(): path.read_bytes()
for path in sorted(bundle.rglob("*"))
if path.is_file()
}
assert okf_corpus_run.main(argv) == 0
second = {
path.relative_to(bundle).as_posix(): path.read_bytes()
for path in sorted(bundle.rglob("*"))
if path.is_file()
}
assert second == first
def test_the_log_link_holds_on_the_unsegmented_path_too(tmp_path: Path) -> None:
"""The two run modes reproject through different code, so both are pinned.
A run without `--plans-dir` uses `STRUCTURED_V1`, whose index is not
per-directory and is rewritten by the singular reprojection rather than the
per-directory one. The harness writes `log.md` in both modes, so a link
that only held on the segmented path would leave the plainer bundle with
exactly the orphan this closes -- and if that path kept the line instead of
dropping it, the second run would carry two.
"""
root = corpus(tmp_path, {"a.md": SUBSTANTIVE, "b.md": SUBSTANTIVE})
bundle = tmp_path / "bundle"
argv = [
"--corpus",
str(root),
"--report",
str(tmp_path / "r.md"),
"--bundle",
str(bundle),
"--ingested-at",
INGESTED_AT,
]
assert okf_corpus_run.main(argv) == 0
index = (bundle / "index.md").read_text(encoding="utf-8")
assert "](log.md)" in index
assert okf_corpus_run.main(argv) == 0
assert (bundle / "index.md").read_text(encoding="utf-8") == index

54
tools/okf_corpus_run.py
View file

@ -68,6 +68,12 @@ __all__ = [
HARNESS_ID = "okf-corpus-run"
# The log's name and title in ONE place, because two of them now read it: the
# file's own frontmatter and the root index entry that points at it. Two
# literals would let the link's label drift away from the thing it labels.
LOG_NAME = "log.md"
LOG_TITLE = "Corpus run history"
def is_degenerate(text: str) -> bool:
"""Zero characters after stripping whitespace. The whole rule, in one line.
@ -196,10 +202,10 @@ class CorpusReport:
lines = [
"---",
"type: Log",
"title: Corpus run history",
f"title: {LOG_TITLE}",
"---",
"",
"# Corpus run history",
f"# {LOG_TITLE}",
"",
f"## {self.ingested_at[:10]}",
"",
@ -369,6 +375,47 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
return parser.parse_args(argv)
def link_log_in_root_index(bundle: Path, profile: BundleProfile) -> None:
"""Point the root index at the log, so the walk section 8 supports reaches it.
Measured on the K2 artifact: the bundle carried a conformant root `log.md`
that no index named, so a consumer entering at `index.md` never reached the
one file carrying `N`.
A LOCAL choice, not a conformance requirement, and the difference is worth
stating rather than implying. Section 9 puts `log.md` at any level and
section 8 has an index enumerate its directory's contents, but upstream's
own bundles do not link it: measured at `9a15b13`, 0 of the 24 shipped
`index.md` files name the single `log.md` in the set. Upstream therefore
shows the link is not REQUIRED -- not that it is disallowed.
It belongs to the harness and not the library. The log's content IS the
run's outcome, so it cannot exist when the indexes are projected; an index
that enumerated it off the directory would gain the link only from the
second run onward and break rebuild-equals-incremental, the property the
segmented bundle is built on. Writing it after the log instead keeps both
runs identical.
THE MEMBERSHIP TEST IS LOAD-BEARING, and measured rather than assumed: the
two reprojections do not treat this line the same way. The per-directory
one drops every managed line before re-emitting its block, so the link is
gone by the time this runs. The flat one keeps a managed line whose target
is not an owned concept -- deliberately, because claiming somebody else's
link on the strength of a regex would delete curated content -- so `log.md`
survives there. Appending unconditionally therefore doubled the entry on
the second unsegmented run. Re-writing the line only when it is absent is
idempotent under both, without either side having to know about the other.
"""
index_path = bundle / profile.index.name
if not index_path.is_file():
return
body = index_path.read_text(encoding="utf-8")
link = profile.index.render_link(LOG_TITLE, LOG_NAME) + "\n"
if link in body.splitlines(keepends=True):
return
index_path.write_text(body + link, encoding="utf-8", newline="")
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
if not args.corpus.is_dir():
@ -422,7 +469,8 @@ def main(argv: list[str] | None = None) -> int:
# harness's output directory would leave the bundle exactly as unverifiable
# as it was before.
bundle.mkdir(parents=True, exist_ok=True)
(bundle / "log.md").write_text(report.render_log(), encoding="utf-8", newline="")
(bundle / LOG_NAME).write_text(report.render_log(), encoding="utf-8", newline="")
link_log_in_root_index(bundle, profile)
print(report.render())
if report.unaccounted or report.merged + report.rejected != report.n:
print(