open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity
llm-ingestion-okf/README.md
Kjell Tore Guttormsen 658b7aafe0 feat(extract): implement pdf behind the [extract] extra with pdfplumber 
Order G2a. Populates the optional `[extract]` extra for the first time with
one parser, `pdfplumber>=0.11.10,<0.12` (MIT), and wires `pdf` through it.
The default install is untouched: exactly one runtime dependency, stdlib
otherwise, enforced by test_packaging.py.

The gate for `pdf` becomes an import probe rather than a frozenset membership
test, exactly as extract.py's docstring had promised. The rejection does not
change: without the extra, `pdf` still raises `extractor_extra_missing` with
the same message. That behaviour is asserted UNCONDITIONALLY via a sys.modules
monkeypatch, so it holds on machines where the parser is installed too — a
skip would have preserved nothing there. Verified in a clean venv without the
extra: 589 passed, 7 skipped; with it, 596 passed.

`docx`/`xlsx` are unchanged and still fail fast — the extra names exactly what
it ships.

The parser choice was forced by measurement, not preference (b73dd9d,
docs/2026-08-21-g2-pdf-extraction-measurement.md): on a real requirement table
pdfplumber keeps 4 of 4 rows with label and value on one line, where pypdf,
pdfminer.six and pymupdf each keep 0 of 4. pymupdf is additionally out on
licence (AGPL-3.0), which an MIT package must not push onto a consumer.

Three facts from that measurement are now carried in code rather than in a
report:

- Extracted text is pinned to an exact transitive parser version
  (pdfplumber pins pdfminer.six==20260107; date-stamped, no stability
  contract). tests/test_extract.py freezes the expected text of a committed
  hand-written fixture so a parser upgrade breaks something visible instead of
  drifting silently. Reasoning at the declaration site and in
  tests/fixtures/README.md.
- Determinism within a version is now held by a test, not only measured once.
- Drawn content does not survive extraction. Every pdf extraction emits the
  new `ExtractionWarning`: figures have no text to recover, so a bundle built
  from drawn documents is incomplete by construction. Stated categorically
  rather than detected — deciding "is there a figure here" is the layout
  heuristic G2b declined.

Two new error codes, both mirroring existing patterns: `extractor_empty_pdf`
(a scanned/image-only PDF, refused rather than persisted as an empty concept)
and `extractor_pdf_error` (parser failure wrapped, never leaked).

Structured table recovery (G2b) is NOT implemented and is documented as out of
scope: two independent parsers return the same wrong shape, so the breakage is
document geometry, not a library choice. PDFs enter as prose.

Also corrects an install promise this change would otherwise have published:
the README no longer presents a bare `pip install 'llm-ingestion-okf[extract]'`
as working, because the package is not on an index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtNhsdHnMGtMi7U2mvMU8z
2026-08-21 20:22:39 +02:00

291 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# llm-ingestion-okf
Shared OKF (Open Knowledge Format) ingestion library: spec-based connectors, bundle inbox, and external-bundle import. Security delegated to llm-ingestion-guard.
Status: phases 1–3 are implemented. Phase 1 (spec-based ingestion) covers
manifest validation, the `file`/`sql`/`http` connectors, deterministic
materialization, index generation, and the golden fixture suite under
`examples/`. Phase 2 adds the bundle inbox (`process_inbox`) and
external-bundle import (`import_bundle`), both against an **injected** persist
gate, with `llm_ingestion_okf.guard_adapter` wiring that gate to the real
guard (see below). Phase 3 makes the bundle contract configurable, so types,
layers, frontmatter sets, index shape, and reserved-file policy are carried by
a profile rather than by constants (see [Upstream OKF
versions](#upstream-okf-versions)). Binary extraction is partial: `pdf` is
implemented behind the optional `[extract]` extra, while `docx`/`xlsx` remain
unimplemented and are rejected fail-fast. Phase 4 (the Node half) is planned
(see `docs/plan/`).
## Install
Python 3.10+. Neither this package nor the guard it depends on is on a package
index yet. With uv, one command is enough:
```
uv pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.4.0"
```
uv resolves the guard on its own, because it reads the `[tool.uv.sources]`
entry in the `pyproject.toml` **of the tag it is installing**, and `v0.4.0`
points that entry at the guard tag below. Measured 2026-07-25 and re-measured
2026-08-20 with an empty `uv` cache; both runs installed
`llm-ingestion-guard==0.2.0` + `llm-ingestion-okf==0.4.0` and imported clean.
With plain pip, the transitive git dependency does not resolve on its own —
**install the guard first**, or installing this package fails with
`No matching distribution found for llm-ingestion-guard`:
```
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"
pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.4.0"
```
The guard tag is paired to the okf tag, not to this branch: `v0.4.0` declares
`llm-ingestion-guard>=0.2,<0.3`, which `v0.2.0` satisfies and later guard tags
do not. `main` has since moved its own pin to `>=0.3,<0.4` (see
[Requirements](#requirements)); that pin reaches you in the next stable tag,
not in the commands above. Reading a pin off this branch and installing it
against `v0.4.0` is the one combination that fails.
`v0.4.0` is the current stable tag. `v0.5.0a2` is a pre-release for the named
OKF v0.2 pilot set only; pin it only if you are one of them (see
[Upstream OKF versions](#upstream-okf-versions)).
## Planned scope (v1)
The library provides three entry points for getting content into an OKF
bundle:
1. **Spec-based ingestion.** An implementation of the normative ingest
specification owned by `portfolio-optimiser-commons`: manifest →
`file`/`sql`/`http` connector → deterministic materialization of
`ingest-{id}.md` concept files → index generation. Zero model calls in the
run path; output is reproducible byte-for-byte against golden fixtures.
2. **Bundle inbox.** A drop directory where common file types are converted
to OKF concept files. All file-type→text extraction lives in this library:
`md`, `txt`, `csv`, `json`, and `html` are handled by the stdlib core;
`pdf` requires the optional `[extract]` extra and is rejected fail-fast
without it; `docx` and `xlsx` ship no parser yet and are always rejected.
Extracted text passes the security gate before anything is persisted.
3. **External bundle import.** Import and merge of third-party OKF bundles:
each concept is assessed via the security gate, and only concepts that
pass are merged, materialized, and linked into the index.
## Boundary: security is delegated
Security is owned by the sibling package
[`llm-ingestion-guard`](https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security)
(pinned `>=0.3,<0.4`). The division is strict:
- **guard** answers "is this content safe to persist?" — scan, sanitize,
quarantine, fail-secure, provenance stamping.
- **this library** does the plumbing — connect a source, materialize a
deterministic OKF bundle, generate the index.
No security functionality is reimplemented here.
### What is gated today: read this before trusting a door
- **Door A (`materialize_bundle`) is ungated.** It calls nothing before
writing to disk and writes what it is given. A caller materializing
untrusted content is responsible for gating it.
- **Doors B and C (`process_inbox`, `import_bundle`) gate through an adapter
you pass in.** Each takes a `gate` argument; the flow hands it the content
and obeys the verdict, refusing to persist anything that does not clear the
guard's non-blocking floor — including a disposition it does not recognise,
and (at Door C) a concept the gate returned no verdict for. What it cannot
do is check that your adapter is a real guard: a permissive stub approves
everything, and the flow will believe it.
`llm_ingestion_okf.guard_adapter` is the adapter over the real guard, and the
only module here that imports it — importing the package itself does not:
```python
from llm_ingestion_okf import process_inbox
from llm_ingestion_okf.guard_adapter import inbox_gate
result = process_inbox(inbox_dir, bundle_dir, "2026-07-25T12:00:00Z",
okf_type="reference", gate=inbox_gate)
```
Two properties of that adapter are worth knowing before you rely on it.
It screens the **exact bytes it persists** — the guard's `prepare_input`
bookend prepares text for a model call, which this library never makes, so
only `screen_output` is used and the screened string is the written string.
And it **refuses rather than repairs**: a file carrying an invisible
zero-width or bidi character is rejected, not silently stripped and written.
Door B screens under the untrusted-upload policy, so any finding at all is
held back rather than persisted.
This section is stated plainly because earlier wording ("calls the guard at
every persist gate") described the intended end state in the present tense,
and a consumer reasonably read it as safe-by-default.
## Roadmap
The library is built in four phases so that every known OKF surface in the
ecosystem is eventually covered. Each phase has a detailed plan with
verification criteria:
1. Spec-based ingestion (Python) with byte-exact golden fixtures —
[plan](docs/plan/phase-1-door-a.md).
2. Bundle inbox and external-bundle import (Python), guard-gated —
[plan](docs/plan/phase-2-doors-b-c.md).
3. Configurable bundle contract (types, layers, frontmatter sets, index
shape, and reserved-file policy as configuration), enabling stricter
bundle profiles such as `strict-v1` —
[plan](docs/plan/phase-3-configurable-contract.md).
4. A `node/` half: a zero-dependency Node/ESM package (importable and
CLI-invokable, vendored per consumer) providing bundle checking, index
generation, inbox processing, and document conversion for the OKF
second-brain plugin ecosystem. The Python and Node halves share the OKF
contract and fixture suite, not code —
[plan](docs/plan/phase-4-node-half.md).
## Upstream OKF versions
The library targets the current latest version of Google's OKF. Support is
**additive** — a new upstream version arrives as a new profile, never as a
migration of an existing one — so an *upstream* release does not change the
bytes an existing profile emits.
That guarantee is about upstream, and one profile tracks a second contract as
well. `DEFAULT` states the ingest-spec owned by `portfolio-optimiser-commons`,
so when they change that spec, `DEFAULT` follows them. It happened on
2026-08-09: `generated` moved from `true` to
`{ by: process:okf-ingest, at: <ingested_at> }`, one changed line per generated
file. Upgrading across it costs a re-run and nothing more — a profile still
recognises bundles stamped by earlier versions, so re-running writes in place
instead of refusing. `DEFAULT` remains OKF v0.1 on every axis upstream owns.
| Profile | Contract | Status |
|---|---|---|
| `DEFAULT` | commons' ingest-spec layer (OKF v0.1 semantics) | stable |
| `STRICT_V1` | a consumer's ratified v0.1 contract | stable |
| `OKF_V0_2` | OKF v0.2 | **provisional**, pre-release only |
| `OKF_LATEST` | alias for the latest version supported as *stable* | currently `DEFAULT` |
`OKF_V0_2` ships first as a pre-release to a named pilot set and may change on
their feedback without a deprecation cycle. Pin the versioned constant rather
than `OKF_LATEST` unless you have explicitly opted into tracking; `OKF_LATEST`
moves at general availability, which is a deliberate release event rather than
a side effect of an upgrade.
Selecting a profile is keyword-only, so existing call sites are unaffected:
```python
materialize_bundle(manifest, bundle_dir, ingested_at, profile=OKF_V0_2)
```
A bundle may declare the version it targets. OKF v0.2 §12 makes this a MAY, and
puts the declaration in the bundle-root `index.md`'s frontmatter block. The
profile names the key; the **caller supplies the value**, because that value
tracks the upstream version and is not this library's to decide:
```python
materialize_bundle(
manifest, bundle_dir, ingested_at,
profile=OKF_V0_2,
root_frontmatter_values={"okf_version": "0.2"},
)
```
Omit the argument and no frontmatter block is written. Offering a key the
profile does not name is refused before anything is written to disk.
### Attested computations (v0.2 §10)
`OKF_V0_2` supports the `Attested Computation` type as a **format**: its five
contract fields — `runtime`, `parameters`, `computation`, `executor`,
`attester` — are emitted in canonical position, judged, and round-tripped.
`runtime` is required for that type and for no other, which the profile
expresses through `FrontmatterSchema.required_by_type`; a type the mapping does
not name carries no extra requirement, because §14 forbids a consumer to reject
on an unknown `type`.
Nothing here executes a computation or checks an attestation. Upstream defers
the receipt and verdict wire formats, so there is no contract to implement, and
the question an attestation answers — was this value produced the sanctioned
way — is not this library's. It re-enters scope when upstream specifies the
protocol.
On the import side, a third-party concept may name an `executor` or `attester`
resource pointing at executable code. Door C imports the **pointer** and never
the code — it writes concepts verbatim and skips every non-`.md` file — so such
a reference may not resolve, or may resolve to a file the destination tree
already holds under that path. Each one is reported in
`ImportResult.unverified_references`; the concept still merges, because §14
forbids rejecting a bundle over a broken cross-link while §10.5 asks a consumer
to surface rather than silently drop. The report names the pointer key, not the
resource it points at: recovering the resource needs the structured reader.
One limit worth knowing before you write such a concept: §10.2 presents
`executor` and `attester` as nested block mappings, and this library's
frontmatter parser is line-oriented. It reads inline **flow** mappings
(`executor: { resource: …, receipt: [ … ] }`) as opaque values that round-trip
unchanged, but it cannot read the block form — two block mappings that both
carry a `resource` collapse into one namespace and the first is lost. Write the
flow form; both are valid YAML, and a real YAML consumer recovers the same
structure from either.
## Non-goals
- Verdict/feedback machinery from the method specification (stays in the
consuming repositories).
- Embedding- or retrieval-layer functionality.
- Security functionality, in either runtime — that is always
`llm-ingestion-guard`'s domain.
## Requirements
Python 3.10+, and exactly one runtime dependency — the security boundary,
`llm-ingestion-guard>=0.3,<0.4`. Everything else is stdlib. The commands are
under [Install](#install); what follows is why they look the way they do.
From a checkout, the test suite runs with:
```
.venv/bin/python -m pytest
```
The suite is the verification surface for everything above: 589 tests, run on
2026-08-13 against this branch. It is not shipped in an installed
distribution — `tests/` lives at the repository root, so this command needs a
clone rather than a `pip install`.
A git URL is a PEP 508 direct reference and pins one exact tag, so it is an
install-time *channel*, not the pin: the range above stays the declared
dependency — a wheel built from this branch carries `Requires-Dist:
llm-ingestion-guard<0.4,>=0.3`, measured 2026-08-10 — and resolves normally
once the package index exists. A wheel built from a *tag* carries that tag's
range instead, which is why the install commands pair tag with tag.
The optional `[extract]` extra ships one parser, `pdfplumber` (MIT), for `pdf`;
`docx`/`xlsx` are still unimplemented. It is opt-in because it pulls binary
wheels (`pillow`, `pypdfium2`), which the default install must never do.
Request it by appending `[extract]` to the package name in whichever install
command from [Install](#install) you are using — this package is not on an
index, so a bare `pip install 'llm-ingestion-okf[extract]'` does **not** work
today, and the error message naming that command is written for the day it
does. The extra is unreleased: it reaches a consumer through a tag that
contains it, and no such tag exists yet.
Two properties of the extra are worth knowing before depending on its output:
- **Extracted text is pinned to an exact parser version.** `pdfplumber` pins
`pdfminer.six==20260107` exactly, and `pdfminer.six` ships date-stamped
releases with no stability contract. Extraction is deterministic within a
parser version and not guaranteed across one, so a golden fixture built on
extracted PDF text is a fixture migration away from any parser upgrade.
- **Text extraction recovers text, and nothing that is drawn.** Figures,
diagrams and images have no text to recover — only their captions survive —
so a bundle built from drawn documents is incomplete by construction. The
library says so itself: every `pdf` extraction emits an `ExtractionWarning`.
Structured table recovery is separately out of scope; PDFs enter as prose.
The planned Node half targets Node/ESM with zero npm dependencies.
## License
MIT — see [LICENSE](LICENSE).
Reference in a new issue View git blame Copy permalink