Kjell Tore Guttormsen
b9372ad7e8
`_pandoc.py` hands back a converter whose identity is known, or refuses. The wheel is not enough on its own. pypandoc searches PATH before its own bundled binary and keeps the highest version found, so on this host the vendored 3.9 was silently bypassed for the system 3.10.2 -- measured a third time before writing this. The resolver reads the installed package's own `files/pandoc` path and asserts the reported version against a frozen PANDOC_VERSION literal, raising `extractor_binary_version` naming both, `extractor_binary_missing` when the wheel carries no binary, and `extractor_extra_missing` when the extra is absent. A mismatch is refused rather than used with a warning: extraction is deterministic within a converter version and not across one, and a byte-pinned fixture cannot tell "a different converter ran" from "we introduced a defect". Two defects found by measuring rather than by the suite: 1. The first implementation asked `pypandoc.get_pandoc_version()`, which answers from a module global that `clean_pandocpath_cache()` does not reset. It therefore reported whichever binary was probed FIRST in the process -- 3.10.2 for the bundled 3.9 binary. The suite was green because nothing in it probed the host binary first. Now `_get_pandoc_version(path)` probes the argument, with no cache and no search in the way, and a regression test poisons the cache before resolving. Negative control: that test fails on the old mechanism. 2. The module docstring named the process-spawning API in prose, which is enough to fail the model-free gate -- the gate is a grep. Reworded. The gate now proves the narrower "no model vendor is reachable from src/", stated in the module rather than glossed. os.environ is restored on both the success and the failure path, and a pre-existing override is put back rather than deleted. Suite 887 -> 895. mypy --strict clean (pypandoc joins the guard's ignore_missing_imports override; every value it returns is coerced here). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
139 lines
6.9 KiB
TOML
139 lines
6.9 KiB
TOML
[build-system]
|
|
requires = ["hatchling"]
|
|
build-backend = "hatchling.build"
|
|
|
|
[project]
|
|
name = "llm-ingestion-okf"
|
|
version = "0.5.0a2"
|
|
description = "Shared OKF (Open Knowledge Format) ingestion library: spec-based connectors, bundle inbox, and external-bundle import, with security delegated to llm-ingestion-guard."
|
|
readme = "README.md"
|
|
license = "MIT"
|
|
requires-python = ">=3.10"
|
|
authors = [{ name = "Kjell Tore Guttormsen" }]
|
|
classifiers = [
|
|
"Development Status :: 3 - Alpha",
|
|
"Intended Audience :: Developers",
|
|
"Operating System :: OS Independent",
|
|
"Programming Language :: Python :: 3",
|
|
"Programming Language :: Python :: 3.10",
|
|
]
|
|
# Exactly one runtime dependency, ever: the security boundary. Everything
|
|
# else is stdlib. The version range is the real pin — it resolves normally
|
|
# against a package index, and is satisfied today by the git+https tag
|
|
# install documented in the README (a direct reference is an install-time
|
|
# channel, not a dependency declaration).
|
|
#
|
|
# Floor 1.2, not the 1.0.0 freeze: this library needs the flow-mapping
|
|
# frontmatter support (`generated: { by: x, at: y }`) that landed in the
|
|
# guard's 1.2.0, without which Door C fail-secures every concept carrying
|
|
# it. Ceiling <2.0, not a narrower minor: the guard's own 1.0.0 release
|
|
# promises no exported name is removed, renamed or given a different
|
|
# meaning short of a 2.0.0 — calibration (severities, dispositions) is
|
|
# explicitly free to move within 1.x under that same promise, so a tighter
|
|
# ceiling here would claim a stability guarantee the guard does not need to
|
|
# keep and we do not need to demand.
|
|
dependencies = ["llm-ingestion-guard>=1.2,<2.0"]
|
|
|
|
[project.optional-dependencies]
|
|
# Binary file-type extraction parsers. OPT-IN ONLY: this extra pulls binary
|
|
# wheels (pillow, pypdfium2) and a transitive tree that core must never have —
|
|
# the "exactly one runtime dependency" rule above covers the default install,
|
|
# and this extra is outside it by construction.
|
|
#
|
|
# The extra names the parsers it actually ships, so a consumer installing it
|
|
# gets what the error message promised and nothing else. It ships two: a `pdf`
|
|
# reader, and a converter that reaches the office types.
|
|
#
|
|
# WHY pdfplumber, and why the floor is not free (measured 2026-08-21,
|
|
# docs/2026-08-21-g2-pdf-extraction-measurement.md): on a real Vegnormalene
|
|
# requirement table pdfplumber keeps 4 of 4 rows with label and value on the
|
|
# same line; pypdf, pdfminer.six and pymupdf each keep 0 of 4, emitting all
|
|
# labels then all values, which a downstream reader can only re-pair by
|
|
# guessing. In a `krav` document that is a wrong answer that looks right.
|
|
# pymupdf is additionally out on LICENSE (AGPL-3.0 or commercial) — this
|
|
# package is MIT and an extra must not hand a consumer copyleft they did not
|
|
# choose.
|
|
#
|
|
# PARSER VERSION IS PART OF THE OUTPUT CONTRACT. pdfplumber pins
|
|
# `pdfminer.six==20260107` exactly, and pdfminer.six ships date-stamped
|
|
# releases with no stability contract. Extraction is deterministic WITHIN a
|
|
# parser version (measured, 5 configurations) and NOT guaranteed across one.
|
|
# `tests/test_extract.py` holds that promise against a committed fixture, so
|
|
# widening this range makes a test go red instead of letting extracted text
|
|
# drift silently. See tests/fixtures/README.md.
|
|
#
|
|
# WHY THE CONVERTER BINARY IS VENDORED RATHER THAN FOUND ON PATH. The `xlsx`
|
|
# and `pptx` readers exist only from pandoc 3.8.3. Debian 12 ships 2.17.1.1
|
|
# and Ubuntu 24.04 ships 3.1.3, so a PATH binary cannot deliver two of the
|
|
# five office formats on current stable distributions -- and a library whose
|
|
# output depends on which pandoc a host happens to carry is not deterministic
|
|
# in the sense the rest of this package means it.
|
|
#
|
|
# `pypandoc-binary` carries the binary inside the wheel (7 platform wheels at
|
|
# 1.17, including macosx x86_64/arm64, manylinux and musllinux x86_64/aarch64,
|
|
# and win_amd64 -- measured on the PyPI JSON API 2026-09-02). The pin is
|
|
# EXACT, not a range, because the binary's version is part of the output
|
|
# contract in the same way pdfminer.six's is: extraction is deterministic
|
|
# within a converter version and not across one.
|
|
#
|
|
# This does not widen the runtime dependency surface. The rule above governs
|
|
# `project.dependencies`, which still names the guard alone; the extra is
|
|
# outside it by construction, and the test below now pins its contents so a
|
|
# third entry cannot arrive unexamined.
|
|
extract = ["pdfplumber>=0.11.10,<0.12", "pypandoc-binary==1.17"]
|
|
|
|
[dependency-groups]
|
|
dev = ["pytest>=8", "mypy>=1.14", "ruff>=0.9"]
|
|
|
|
[tool.hatch.build.targets.wheel]
|
|
packages = ["src/llm_ingestion_okf"]
|
|
|
|
[tool.ruff]
|
|
line-length = 100
|
|
target-version = "py310"
|
|
|
|
[tool.mypy]
|
|
strict = true
|
|
python_version = "3.10"
|
|
|
|
# llm-ingestion-guard ships no py.typed marker, so its symbols arrive as Any.
|
|
# The adapter coerces every value it carries across the seam to a concrete
|
|
# type, which is what keeps --strict meaningful on this side of it.
|
|
[[tool.mypy.overrides]]
|
|
module = ["llm_ingestion_guard", "llm_ingestion_guard.*"]
|
|
ignore_missing_imports = true
|
|
|
|
# `pypandoc` ships no py.typed marker either. Only `_pandoc.py` imports it, and
|
|
# every value it hands back is coerced to `str`/`Path` there before it reaches
|
|
# the rest of the package -- the same discipline as the guard adapter above.
|
|
[[tool.mypy.overrides]]
|
|
module = ["pypandoc", "pypandoc.*"]
|
|
ignore_missing_imports = true
|
|
|
|
# Install CHANNEL for the guard, which is not on a package index yet. It is
|
|
# uv-specific, and it reaches further than a dev-only setting: a consumer
|
|
# installing this package from git WITH UV picks the guard up from this tag
|
|
# automatically, because uv reads this file when it builds from the source
|
|
# tree. Measured against an empty cache 2026-07-25, 2026-08-20, and
|
|
# 2026-08-21 on uv 0.9.8. The 08-21 run also measured the TRANSITIVE form: a
|
|
# separate consumer project naming only this package still resolves the guard
|
|
# from the entry below, because this package reaches it as a git source.
|
|
#
|
|
# That source is the whole reach. A wheel carries Requires-Dist and nothing
|
|
# else, so this entry cannot survive an index install — and while the guard is
|
|
# off-index, removing it would break the one-command uv path the README
|
|
# documents.
|
|
#
|
|
# pip does not read it at all: it resolves [project.dependencies] alone and
|
|
# fails with "No matching distribution found for llm-ingestion-guard" until
|
|
# the guard is installed from its own tag first (README; measured 2026-08-21,
|
|
# both the failure and the two-command recovery).
|
|
#
|
|
# Either way the range above stays the pin, and the pin is per-tree: a wheel
|
|
# built from THIS tree carries `Requires-Dist: llm-ingestion-guard<2.0,>=1.2`,
|
|
# measured 2026-08-23 against the built wheel. The `<0.4,>=0.3` this comment
|
|
# carried before was the `v0.3.4` tag's range — still true of that tag, never
|
|
# true of this tree. Reading a range off one and installing it against the
|
|
# other is the one combination that fails.
|
|
[tool.uv.sources]
|
|
llm-ingestion-guard = { git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git", tag = "v1.2.0" }
|