feat(cli): okf build, one installed command for folder in, bundle out

Until now "run the door over a folder" was a shell loop over two scripts
under `tools/`, with nine flags between them and a `--path-prefix` rule
that lived in a code block in a measurement report. Neither script was
packaged (`pyproject.toml` ships `src/llm_ingestion_okf` only), so the
path the published K1/K2 numbers were measured on was reachable from a
clone and nowhere else.

`okf build <folder> --bundle <dir>` is that path, packaged, declared as a
console script and installed with the wheel. It is orchestration only:
the proposer and the corpus harness MOVED into the package
(`llm_ingestion_okf.propose`, `llm_ingestion_okf.corpus`) and the two
`tools/` scripts became thin entry points to them, so the published
reproduction blocks still run and there is exactly one implementation of
each rule. Neither move adds a dependency or a model call.

Two decisions belong to this layer and are stated where they are made.
A document's proposed paths are scoped by its RELATIVE PATH minus the
extension, not its basename: the door walks recursively now, and two
documents named alike in different folders would otherwise collide on a
path Door B is supposed to make impossible rather than merely detect.
And omitted timestamps do not come from the clock -- `--ingested-at` and
`--proposed-at` default to one shared epoch constant, because a
wall-clock default would put a changing byte in the artifact and take
rebuild-equals-incremental away from every caller who did not pass them.

Arm C and Arm D stay off and are not exposed here.

Measured on the 43-file K2 corpus, one invocation against the two-script
bundle of 2026-09-03: N = 43 computed, merged 39/43, coded rejections
4/43 (`extractor_unknown` 3, `extractor_empty_pdf` 1), K1b 39 + 4 = 43,
exit 0, 779.43 s. 1107 of 1108 files byte-identical. The one that
differs is the root `index.md`, by exactly the `log.md` link a commit
fifteen hours younger than the stored artifact adds -- appending that
line to the stored file reproduces the new one byte for byte. Against
the two scripts at THIS commit the trees agree in full, which is what
the byte-identity test holds.

Suite 1127 passed after `git add` (1113 before), mypy --strict clean,
ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 05:06:33 +02:00
commit 4ce14ae5dd
15 changed files with 2042 additions and 1244 deletions

View file

@ -52,6 +52,56 @@ against `v0.4.0` is the one combination that fails.
OKF v0.2 pilot set only; pin it only if you are one of them (see
[Upstream OKF versions](#upstream-okf-versions)).
## Build
Installing the package installs one command. A folder of documents in, an OKF
bundle out:
```
okf build ./documents --bundle ./bundle --bundle-id my-bundle --okf-version 0.2
```
It walks the folder recursively, proposes a segmentation for each document with
the mechanical rules, replays those proposals through the bundle inbox, writes
the bundle and its `log.md`, and prints the run's numbers. Every proposal is
marked `PROPOSED` and `adjudicated: false` — the command segments nothing a
human has approved, and says so in the artifact.
The last line that matters is the conservation identity: `merged + coded
rejections == N`, where `N` is the folder's file count read at run time. **The
run exits non-zero when it does not hold**, and names the unaccounted files, so
a pipeline cannot mistake a partial bundle for a complete one.
Flags worth knowing: `--segments off` ingests each document as one concept and
asks for no root values; `--plans-dir` keeps the proposals instead of
discarding them; `--report` writes the full report to a file as well as stdout.
`--ingested-at` and `--proposed-at` default to `1970-01-01T00:00:00Z` rather
than the clock, so two builds of the same folder are byte-identical — a
wall-clock default would break rebuild-equals-incremental for every caller who
did not pass them.
Measured 2026-09-07 on a 43-file corpus (33 `pdf`, 5 `docx`, 2 `xlsx`, and
three files no reader accepts), one
`okf build` invocation replacing the shell loop over `tools/` that produced the
same corpus's bundle on 2026-09-03:
| figure | value |
|---|---|
| `N` (folder file count, computed) | 43 |
| merged | 39/43 |
| coded rejections | 4/43 (`extractor_unknown` 3, `extractor_empty_pdf` 1) |
| K1b | `39 + 4 = 43 = N`, exit `0` |
| files written | 1108 |
| identical to the 2026-09-03 bundle | 1107/1108 |
| wall time | 779.43 s total, 18.126 s per file |
The one file that differs is the root `index.md`, by exactly one line: the link
to the bundle's own `log.md`, added by a commit that postdates the stored
artifact by fifteen hours. Appending that line to the stored `index.md`
reproduces the new one byte for byte. The command's own byte-identity test
compares it against the two scripts at the current commit, where the two agree
over the whole tree.
## Implemented scope (v1)
The library provides three entry points for getting content into an OKF