open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(build): name the documents the gate refused whole, and close as 1.0.0

Browse source 
Measured 2026-09-20 on an official documentation corpus of 594 sources built
with the shipped default gate `guard-trusted-source`: 17 sources were refused
OUTRIGHT -- `fail_secure` 3, `quarantine_review` 14 -- and 16 of them were
among 197 official documentation pages, the pages on hooks, skills,
permissions, errors, env-vars and authentication among them. The summary said
only `fail_secure`: 3/594. Three of the four facts a reader needs were missing:
the COUNT of documents the gate dropped (the existing `rejected (coded)` line
sums gate refusals and extraction failures, two failures with two different
remedies), the NAMES, and the way out. Rebuilt with `--gate none`, all 17 went
through untouched, so the refusal is the gate and not the readers.

`okf build` now prints a `Documents the gate refused WHOLE` section directly
under the denominator, carrying all four: the count with its denominator, the
names capped at ten with the rest in the bundle's `log.md`, the codes, and
`--gate none` for a source you vouch for yourself. The same fact goes to stderr
in one line, built from the same field, because `okf build > report.txt` is an
ordinary thing to do. `log.md` gains one bullet naming every refused document,
uncapped.

The exit code deliberately does not move. The build is valid -- every refusal
is coded, the conservation identity holds, and the bundle is a true record of
what the gate allowed. What was wrong was the silence.

A run the gate refused nothing from is byte-identical in both places, which is
the known-negative in the new suite: no bundle this repository ships was built
with a gate refusal, so this cannot have moved a byte measured here.

Also, and measuring nothing new:

- README gains `Known limitations` high up -- the gate's refusals and the way
  out, the absent ceiling on what one run pays for images (a 70 KB PDF with 16
  images under the declared limit reached 851 MB peak RSS; RLIMIT_AS is not
  enforceable on this platform, so the 512 MiB per-link budget is the whole
  bound), the three gates of this repository that are RED today (retrieval
  5/7/8/9, MCP 2, accounting 2/3/6 -- all three re-run on this commit), what
  the content accounting does not count, and the rough edges nothing is
  planned for.
- The two `pip install` lines under "Install in detail" install `[extract]`.
  The first screen does; those two did not, so the two recipes produced
  different installations and the detailed one reports `resolved converter
  path: unresolved (extractor_extra_missing)`.
- Version `1.0.0`, synced across pyproject, `__version__`, `uv.lock`, the four
  README install lines, the install prose, the current-tag entry and the
  CHANGELOG, where the two "after the 0.10.1 notes were written, untagged"
  sections are folded in. It adds no capability over `v0.10.1`; what it adds
  is that the tool says what it does not do.

Suite: 2325 passed, 2 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-20 21:44:49 +02:00
commit 404bed26e2
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
10 changed files with 512 additions and 69 deletions
Download patch file Download diff file Expand all files Collapse all files

79
README.md
View file

@ -10,7 +10,7 @@ types](#supported-file-types) lists each one with the evidence behind it.
Python 3.10+ and [uv](https://docs.astral.sh/uv/). One line:
```sh
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.1"
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v1.0.0"
```
## Use it
@ -36,6 +36,64 @@ Code](#consume-in-claude-code) for the same thing in steps and with several
bundles at once, [Build](#build) for the flags, [Requirements](#requirements)
for the pip fallback and the guard pairing.
## Known limitations
Read this before pointing the tool at documents you depend on. Every number
here was measured; none of it is a plan.
- **The default gate refuses whole documents, and they are documents you may
want.** Measured 2026-09-20 on an official documentation corpus of 594
sources: `guard-trusted-source`, the shipped default, refused **17 of 594
sources outright** — `fail_secure` 3, `quarantine_review` 14 — and **16 of
those were 197 official documentation pages** (8.1 %), among them the pages
on hooks, skills, permissions, errors, env-vars and authentication. Not one
element of a refused document reaches the bundle. Rebuilt with `--gate none`,
all 17 went through untouched, so the refusal is the gate and not the
readers: a page of official documentation naturally carries commands and
instruction-shaped text, and the guard reads that as something to hold for
review. The build says so now — it names the count, the files, the codes and
that command — and exits 0, because the bundle is a true record of what the
gate allowed. **If you vouch for the source yourself, build with `--gate
none`;** the bundle then records that nothing was screened. The default was
chosen on one measurement over one pinned bundle's 453 concept bodies, which
is a thin denominator for a decision this consequential.
- **Nothing bounds what one run pays for images.** Each decode link is capped
(`MAX_FILTER_DECODE_BYTES`, 512 MiB) and an oversized picture is refused with
its own code, but the run as a whole has no ceiling: measured, a 70 KB PDF
carrying 16 images each under the declared limit reached **851 MB peak RSS**
and every picture was carried. A hard cap outside Python was measured and is
not available here — `resource.setrlimit(RLIMIT_AS)` raises on Darwin 26.6.2
and is not enforced — so the per-link budget is the whole bound.
`--no-assets` takes the image path out entirely.
- **Three of this repository's own gates are RED, and each red row is a stated
finding rather than a bug to be surprised by.** The retrieval gate is red on
rows 5, 7, 8 and 9, the MCP gate on row 2, and the content accounting's judge
on rows 2, 3 and 6. For a user that means: retrieval quality is measured but
not yet green on a held-out set (rows 5, 8), two mechanical mutants of the
ranking survive with 0 ranks and 0 deliveries moved (row 7), no gold set
exists for the K2 corpus (row 9), MCP anchors and concept ids are different
vocabularies so `okf_fetch` cannot be addressed with a set's anchor (row 2),
and the accounting still reports real losses on the reference corpus (rows 2,
3, 6). The rows and their numbers are under [Judge the
retrieval](#judge-the-retrieval-python3-toolsokf_retrieval_gatepy) and
[Serve a bundle over MCP](#serve-a-bundle-over-mcp-okf-mcp).
- **The content accounting counts the element classes its vocabulary names, and
no others.** `0 unaccounted` is a statement about those classes, not about the
document: a file whose suffix has no reader is accounted at file level only,
parts no vocabulary names (headers, footers, endnotes, comments, speaker
notes, cell formulas) are outside it, and an image in `xlsx`, `md`, `txt`,
`csv`, `json`, `odt` or `rtf` is unaccounted and therefore red. It is opt-in
(`--accounting PATH`) for that reason. The full list is under
[Build](#build).
- **A few rough edges, named rather than fixed.** `okf skill` requires `--out`
and exits 2 without it. Over MCP, `okf_describe` requires `bundle_id` where
`okf_ask` treats it as optional. One `okf_ask` answer measured over 300 KB on
a 2 002-concept bundle, because the payload's `withheld` list carries one
entry per concept that was not delivered — the payload itself was well inside
its budget.
- **There is no context graph and no visualisation.** Nothing in this package
draws a bundle.
## What this library is
Status: phases 1–3 are implemented. Phase 1 (spec-based ingestion) covers
@ -120,11 +178,11 @@ Neither this package nor the guard it depends on is on a package index yet, so
both install by direct reference. With uv, one command resolves both:
```sh
uv pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.1"
uv pip install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v1.0.0"
```
uv resolves the guard on its own, because it reads the `[tool.uv.sources]`
entry in the `pyproject.toml` **of the tag it is installing**, and `v0.10.1`
entry in the `pyproject.toml` **of the tag it is installing**, and `v1.0.0`
points that entry at `llm-ingestion-guard` `v1.4.0`. Use `uv tool install`
instead of `uv pip install` when you want the `okf` command on `PATH` without an
active virtualenv — that is the form the first screen shows.
@ -135,10 +193,10 @@ With plain pip, the transitive git dependency does not resolve on its own —
```sh
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.4.0"
pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.1"
pip install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v1.0.0"
```
The guard tag is paired to the okf tag, not to this branch. `v0.10.1` declares
The guard tag is paired to the okf tag, not to this branch. `v1.0.0` declares
`llm-ingestion-guard>=1.2,<2.0`, which `v1.4.0` satisfies; the pairing above is
read off that tag's own `[tool.uv.sources]`, not off this branch. Reading a pin
off `main` and installing it against an older okf tag is the one combination
@ -149,7 +207,14 @@ that fails.
These are not install lines. They record what each earlier tag was, so a reader
who meets one in an older document knows what they are looking at.
- `v0.10.1` — the current tag: the image path of `v0.10.0`, with the two
- `v1.0.0` — the current tag, and the version this tool is finished at. It
adds no capability to `v0.10.1`: a document the gate refuses whole is now
named in the run's own summary with its code and with the one command that
carries it anyway, and the front page states what this tool does not do.
Read [Known limitations](#known-limitations) before you depend on it. After
this tag the library is touched for defects found in its own use, and the
next round is Google OKF v0.3.
- `v0.10.1` — the image path of `v0.10.0`, with the two
defects an independent review found in it closed. A remote `<img src>` or
`xlink:href` is inert text with the address in one code span, never a live
markdown image link, and no longer loses the figure's caption. An image is
@ -1103,7 +1168,7 @@ not need this repository — the first line installs the command, the second
builds the bundle and writes a skill beside it, the third asks.
```sh
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.1"
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v1.0.0"
okf project ~/my-documents
claude
```