open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(cli): okf build, one installed command for folder in, bundle out

Browse source 
Until now "run the door over a folder" was a shell loop over two scripts
under `tools/`, with nine flags between them and a `--path-prefix` rule
that lived in a code block in a measurement report. Neither script was
packaged (`pyproject.toml` ships `src/llm_ingestion_okf` only), so the
path the published K1/K2 numbers were measured on was reachable from a
clone and nowhere else.

`okf build <folder> --bundle <dir>` is that path, packaged, declared as a
console script and installed with the wheel. It is orchestration only:
the proposer and the corpus harness MOVED into the package
(`llm_ingestion_okf.propose`, `llm_ingestion_okf.corpus`) and the two
`tools/` scripts became thin entry points to them, so the published
reproduction blocks still run and there is exactly one implementation of
each rule. Neither move adds a dependency or a model call.

Two decisions belong to this layer and are stated where they are made.
A document's proposed paths are scoped by its RELATIVE PATH minus the
extension, not its basename: the door walks recursively now, and two
documents named alike in different folders would otherwise collide on a
path Door B is supposed to make impossible rather than merely detect.
And omitted timestamps do not come from the clock -- `--ingested-at` and
`--proposed-at` default to one shared epoch constant, because a
wall-clock default would put a changing byte in the artifact and take
rebuild-equals-incremental away from every caller who did not pass them.

Arm C and Arm D stay off and are not exposed here.

Measured on the 43-file K2 corpus, one invocation against the two-script
bundle of 2026-09-03: N = 43 computed, merged 39/43, coded rejections
4/43 (`extractor_unknown` 3, `extractor_empty_pdf` 1), K1b 39 + 4 = 43,
exit 0, 779.43 s. 1107 of 1108 files byte-identical. The one that
differs is the root `index.md`, by exactly the `log.md` link a commit
fifteen hours younger than the stored artifact adds -- appending that
line to the stored file reproduces the new one byte for byte. Against
the two scripts at THIS commit the trees agree in full, which is what
the byte-identity test holds.

Suite 1127 passed after `git add` (1113 before), mypy --strict clean,
ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 05:06:33 +02:00
commit 4ce14ae5dd
15 changed files with 2042 additions and 1244 deletions
Download patch file Download diff file Expand all files Collapse all files

10
tests/test_corpus_run.py
View file

@ -22,14 +22,11 @@ passes silently, every green run above it becomes meaningless.
from __future__ import annotations
import sys
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tools"))
import okf_corpus_run # noqa: E402
from llm_ingestion_okf import corpus as okf_corpus_run
INGESTED_AT = "2026-07-25T12:00:00Z"
@ -248,10 +245,9 @@ SEGMENTABLE = (
def _propose(source: Path, out: Path) -> None:
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tools"))
import okf_propose_segments
from llm_ingestion_okf import propose
assert okf_propose_segments.run(source, out, okf_type="reference", proposed_at=INGESTED_AT) == 0
assert propose.run(source, out, okf_type="reference", proposed_at=INGESTED_AT) == 0
def test_a_plan_directory_makes_the_run_segment_and_mark_every_concept_proposed(