Kjell Tore Guttormsen
404bed26e2
Measured 2026-09-20 on an official documentation corpus of 594 sources built with the shipped default gate `guard-trusted-source`: 17 sources were refused OUTRIGHT -- `fail_secure` 3, `quarantine_review` 14 -- and 16 of them were among 197 official documentation pages, the pages on hooks, skills, permissions, errors, env-vars and authentication among them. The summary said only `fail_secure`: 3/594. Three of the four facts a reader needs were missing: the COUNT of documents the gate dropped (the existing `rejected (coded)` line sums gate refusals and extraction failures, two failures with two different remedies), the NAMES, and the way out. Rebuilt with `--gate none`, all 17 went through untouched, so the refusal is the gate and not the readers. `okf build` now prints a `Documents the gate refused WHOLE` section directly under the denominator, carrying all four: the count with its denominator, the names capped at ten with the rest in the bundle's `log.md`, the codes, and `--gate none` for a source you vouch for yourself. The same fact goes to stderr in one line, built from the same field, because `okf build > report.txt` is an ordinary thing to do. `log.md` gains one bullet naming every refused document, uncapped. The exit code deliberately does not move. The build is valid -- every refusal is coded, the conservation identity holds, and the bundle is a true record of what the gate allowed. What was wrong was the silence. A run the gate refused nothing from is byte-identical in both places, which is the known-negative in the new suite: no bundle this repository ships was built with a gate refusal, so this cannot have moved a byte measured here. Also, and measuring nothing new: - README gains `Known limitations` high up -- the gate's refusals and the way out, the absent ceiling on what one run pays for images (a 70 KB PDF with 16 images under the declared limit reached 851 MB peak RSS; RLIMIT_AS is not enforceable on this platform, so the 512 MiB per-link budget is the whole bound), the three gates of this repository that are RED today (retrieval 5/7/8/9, MCP 2, accounting 2/3/6 -- all three re-run on this commit), what the content accounting does not count, and the rough edges nothing is planned for. - The two `pip install` lines under "Install in detail" install `[extract]`. The first screen does; those two did not, so the two recipes produced different installations and the detailed one reports `resolved converter path: unresolved (extractor_extra_missing)`. - Version `1.0.0`, synced across pyproject, `__version__`, `uv.lock`, the four README install lines, the install prose, the current-tag entry and the CHANGELOG, where the two "after the 0.10.1 notes were written, untagged" sections are folded in. It adds no capability over `v0.10.1`; what it adds is that the tool says what it does not do. Suite: 2325 passed, 2 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
116 lines
3.3 KiB
Python
116 lines
3.3 KiB
Python
"""Shared OKF (Open Knowledge Format) ingestion library.
|
|
|
|
Three entry doors: spec-based ingestion (manifest -> connector ->
|
|
deterministic materialization -> index), a bundle inbox converting common
|
|
file types to OKF concepts, and import of external OKF bundles.
|
|
|
|
Security is owned by llm-ingestion-guard, never reimplemented here. Note
|
|
what that does and does not mean today: Door A is UNGATED. It calls no guard
|
|
function on the way to disk, so a caller that materializes external or
|
|
otherwise untrusted content is responsible for gating it. Do not read
|
|
"security is delegated" as "safe by default": materialize_bundle writes what
|
|
it is given.
|
|
|
|
Doors B and C are gated by an INJECTED adapter, and `guard_adapter` is the
|
|
one this library ships over the real guard -- the only module here that
|
|
imports it. Importing this package does not import the guard; a Door A
|
|
consumer keeps working whatever state the dependency is in.
|
|
|
|
Door A (spec-based ingestion) public surface: materialize_bundle plus the
|
|
typed error hierarchy rooted in IngestError.
|
|
|
|
Door B (bundle inbox) public surface: process_inbox plus its result types.
|
|
Its persist gate is INJECTED -- the caller supplies a Gate adapter over
|
|
llm-ingestion-guard (guard_adapter.inbox_gate is the shipped one) and
|
|
process_inbox obeys the verdict, making no security decision of its own.
|
|
|
|
Door C (external bundle import) public surface: import_bundle plus its result
|
|
types. Its gate is injected the same way, over the guard's okf.import_bundle
|
|
(guard_adapter.import_gate is the shipped one):
|
|
the caller declares origin and channel at the door, the gate assesses every
|
|
concept, and only concepts clearing the non-blocking floor are merged. Merged
|
|
concepts are written verbatim -- ownership is proven by content identity
|
|
rather than by a stamp, so an occupied name is only ever re-used when the
|
|
bytes already there are identical.
|
|
"""
|
|
|
|
from .errors import (
|
|
ExtractionError,
|
|
ExtractionWarning,
|
|
IngestError,
|
|
ManifestError,
|
|
MaterializationError,
|
|
NetworkGateError,
|
|
RenderError,
|
|
SourceError,
|
|
)
|
|
from .extract import extract_text
|
|
from .importer import (
|
|
BundleDecision,
|
|
FailedConcept,
|
|
ImportDecision,
|
|
ImportGate,
|
|
ImportResult,
|
|
MergedConcept,
|
|
RefusedConcept,
|
|
UnverifiedReference,
|
|
import_bundle,
|
|
)
|
|
from .inbox import (
|
|
BlockedFile,
|
|
FailedFile,
|
|
Gate,
|
|
GateDecision,
|
|
InboxResult,
|
|
PersistedFile,
|
|
SkippedPath,
|
|
process_inbox,
|
|
)
|
|
from .manifest import (
|
|
Extraction,
|
|
FileSource,
|
|
HttpSource,
|
|
Manifest,
|
|
SqlSource,
|
|
load_manifest,
|
|
)
|
|
from .materialize import IngestResult, materialize_bundle
|
|
|
|
__version__ = "1.0.0"
|
|
|
|
__all__ = [
|
|
"BlockedFile",
|
|
"BundleDecision",
|
|
"Extraction",
|
|
"ExtractionError",
|
|
"ExtractionWarning",
|
|
"FailedConcept",
|
|
"FailedFile",
|
|
"FileSource",
|
|
"Gate",
|
|
"GateDecision",
|
|
"HttpSource",
|
|
"ImportDecision",
|
|
"ImportGate",
|
|
"ImportResult",
|
|
"InboxResult",
|
|
"IngestError",
|
|
"IngestResult",
|
|
"Manifest",
|
|
"ManifestError",
|
|
"MaterializationError",
|
|
"MergedConcept",
|
|
"NetworkGateError",
|
|
"PersistedFile",
|
|
"SkippedPath",
|
|
"RefusedConcept",
|
|
"RenderError",
|
|
"SourceError",
|
|
"SqlSource",
|
|
"UnverifiedReference",
|
|
"extract_text",
|
|
"import_bundle",
|
|
"load_manifest",
|
|
"materialize_bundle",
|
|
"process_inbox",
|
|
]
|