open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(guard): bump the pin to >=0.3,<0.4 and pin Door C's allow_reserved=False

Browse source 
Measure first, widen after. The 19-fixture guard-surface suite was re-run
against v0.3.4 in a scratch venv before the range moved, and reproduced the
three deltas measured against v0.3.3 exactly, with none added. v0.3.4 is the
tag pinned rather than v0.3.3 because it shipped first and repairs a quadratic
regex (okf._MD_LINK_RE) that sits on Door C's own call path.

Door C now passes allow_reserved=False explicitly. The guard added the keyword
in the 0.3 line and defaults it True for received bundles, which would merge a
sender's index.md / log.md instead of rejecting them. The override keeps the
unconditional reserved-name refusal committed to before the keyword existed,
and the reason is structural rather than a second opinion on the guard's scan:
Door C generates the merged bundle's index.md from what it merged and writes
every merged concept verbatim, so a sender's index.md would be a second and
irreconcilable claim on one path.

This is not a behaviour change for anyone on the previous pin: under v0.2.0
the keyword did not exist and reserved names were refused by construction.

The floor is >=0.3 and not >=0.2 for a measured reason. allow_reserved is
absent in v0.2.0 and present from v0.3.0 onward, checked across all five tags:
a >=0.2 floor would admit a version that raises TypeError on every Door C
import. That measurement also corrects a recorded premise -- the plan said the
keyword "shipped in v0.3.3", which read the first version we ran the suite
against as the version it was introduced in. The conclusion held; the reason
did not, and the reason is what a future bump would have relied on.

test_door_c_pins_allow_reserved_false_against_the_guards_default locks both
halves: that the guard still defaults True, without which the override is a
no-op that would pass forever over nothing, and that Door C overrides it.

586 tests, mypy --strict clean, goldens byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V2v1hrDhrff2H3y2TNJHkF
This commit is contained in:
Kjell Tore Guttormsen 2026-08-02 21:08:53 +02:00
commit f536e1384d
12 changed files with 156 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

26
CHANGELOG.md
View file

@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Changed
- **The guard pin moved to `>=0.3,<0.4`, resolved against `v0.3.4`.** The
window is widened only after measurement, never before: the 19-fixture
guard-surface suite was run against `v0.3.4` in a scratch venv first, and
reproduced exactly the three deltas measured against `v0.3.3` — no new ones.
`v0.3.4`'s own fixes are regex-complexity repairs, one of them
(`okf._MD_LINK_RE`) on Door C's call path, with no disposition changes.
- **Door C now passes `allow_reserved=False` to `okf.import_bundle`.** The
guard added the keyword in the `0.3.x` line and defaults it `True` for the
received-bundle path, which would merge a sender's `index.md` / `log.md`
instead of rejecting them. Door C overrides it, keeping the unconditional
reserved-name refusal committed to before the keyword existed. The reason is
structural rather than a second opinion on the guard's scan: Door C generates
the merged bundle's `index.md` from what it merged and writes every merged
concept verbatim, so a sender's `index.md` would be a second, irreconcilable
claim on one path.
**This is not a behaviour change for anyone on the previous pin.** Under
`v0.2.0` the keyword did not exist and reserved names were refused by
construction; the explicit argument preserves that outcome across the bump.
A consumer sees the same rejections, with the same reasons, before and after.
## [0.5.0a2] — 2026-07-31
**This is the pre-release the pilots pin. `v0.5.0a1` was tagged and abandoned

6
CLAUDE.md
View file

@ -29,7 +29,7 @@ one boundary rule:
when the bytes there are already identical, never overwritten otherwise.
**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
(pinned `>=0.2,<0.3`) answers "is this content safe to persist?" —
(pinned `>=0.3,<0.4`) answers "is this content safe to persist?" —
scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is
plumbing: connect source → materialize deterministic OKF bundle → generate
index. Never reimplement security; call the guard at persist gates
@ -151,12 +151,12 @@ Phase 4 preconditions (coordination, not unilateral moves):
## Stack
Python 3.10+. Package `llm_ingestion_okf` (src layout, hatchling).
**Exactly one runtime dependency, ever:** `llm-ingestion-guard>=0.2,<0.3`
**Exactly one runtime dependency, ever:** `llm-ingestion-guard>=0.3,<0.4`
(itself zero-dep), landed with the Door B/C persist gates. Everything else is
stdlib, and a packaging test enforces it. Only `guard_adapter.py` imports the
guard; importing the package does not. Install channel until the package
index exists (a direct reference is a channel, not the pin):
`pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"`.
`pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.3.4"`.
Binary extraction parsers live behind the `[extract]` extra only.
Phase 4 adds a `node/` half: Node/ESM with zero npm dependencies

6
README.md
View file

@ -37,7 +37,7 @@ bundle:
Security is owned by the sibling package
[`llm-ingestion-guard`](https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security)
(pinned `>=0.2,<0.3`). The division is strict:
(pinned `>=0.3,<0.4`). The division is strict:
- **guard** answers "is this content safe to persist?" — scan, sanitize,
quarantine, fail-secure, provenance stamping.
@ -192,14 +192,14 @@ structure from either.
## Requirements
Python 3.10+, and exactly one runtime dependency — the security boundary,
`llm-ingestion-guard>=0.2,<0.3`. Everything else is stdlib.
`llm-ingestion-guard>=0.3,<0.4`. Everything else is stdlib.
That guard is not on a package index yet, so **with pip, install it first** —
otherwise installing this package fails with `No matching distribution found
for llm-ingestion-guard`:
```
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.3.4"
pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.4.0"
```

26
docs/plan/execution-order.md
View file

@ -37,7 +37,8 @@ Outcome recorded in STATE before Phase 2 code starts.
### Stage 0 revisited — measuring guard 0.3.1 before widening the pin
The pin held is `>=0.2,<0.3`, which can only ever resolve `v0.2.0`. The guard repo
The pin held at the time was `>=0.2,<0.3`, which could only ever resolve
`v0.2.0` (it has since moved to `>=0.3,<0.4`, resolving `v0.3.4`). The guard repo
gates its own 1.0 on our fixture suite being measured against `v0.3.1`, so this
measurement is owed outward and is that repo's only named blocker. Order is
*measure first, widen after*: widening the range before measuring would ship a
@ -61,9 +62,24 @@ Procedure — do not vary it, each step exists because of a specific failure mod
5. Green → one release: `>=0.2,<0.4`, uv tag `v0.3.1`, CLAUDE.md and README
updated together. Red → no bump, and the finding goes back to the guard.
Until this lands, consumers of v0.4.0 resolve guard 0.2.0 (`pyproject.toml:25`).
That is a silent under-defend: no build fails, so nothing surfaces it. The
release in step 5 is what closes it.
**Done, and step 5's predicted range was wrong — deliberately superseded.** The
bump landed as `>=0.3,<0.4` against tag `v0.3.4`, not `>=0.2,<0.4` against
`v0.3.1`. Two corrections, both forced rather than chosen:
- **The floor had to rise to `0.3`.** Door C now passes `allow_reserved=False`,
a keyword measured absent in `0.2.0` and present from `v0.3.0` onward. A
`>=0.2` floor would let a consumer resolve `0.2.0` and take a `TypeError` on
every Door C import — the range would have admitted a version the code cannot
call. This is the one case where widening past the measured version is not
conservative but broken. The signature was measured across all five tags
rather than inferred from the one the fixtures happened to run against.
- **The tag is the newest measured one, not the first one asked for.** `v0.3.2`,
`v0.3.3` and `v0.3.4` all shipped while this was open. The suite was re-run
against `v0.3.4` before the bump — pinning the tag that was measured is the
whole point of the order, and an older tag would have been measured against a
guard nobody would install.
That also closes the under-defend below: consumers no longer resolve `0.2.0`.
## Phase 2 — Doors B/C (split: guard-independent first)
@ -172,7 +188,7 @@ These are the objective checks that a stage is done and the next may begin:
matches + installable in CI" or an explicit "front-load guard-independent
half; guard integration blocked on <named item>".
2. **Phase 2 → Phase 3:** Phase 1 golden suite still byte-for-byte; `pyproject`
runtime deps == exactly `llm-ingestion-guard>=0.2,<0.3`; persist-gate proof
runtime deps == exactly one range on `llm-ingestion-guard`; persist-gate proof
test green (a fail-secure fixture yields zero new files).
3. **Phase 3 → Phase 4:** golden suite byte-identical under `DEFAULT`
(`git diff --stat examples/` empty for the phase); `STRICT_V1` cross-profile

44
docs/plan/phase-2-doors-b-c.md
View file

@ -3,7 +3,8 @@
Status: approved roadmap phase (see `CLAUDE.md`); details settled here before code.
Depends on: Phase 1 (materialization + index primitives are reused, never duplicated).
This phase adds the library's first — and only permitted — runtime dependency:
`llm-ingestion-guard>=0.2,<0.3`.
`llm-ingestion-guard` (pinned `>=0.2,<0.3` when this plan was written; the
window moved to `>=0.3,<0.4` after measurement — see the settled note below).
## Goal
@ -93,7 +94,7 @@ No scanning, sanitizing, or quarantine logic is implemented here.
`Origin`/`Channel` vocabularies Door C validates, the result fields the
adapters read, and the upload preset's shape.
- **The pin stays a range; the git URL is an install channel.** A PEP 508
direct reference pins one tag and cannot express `>=0.2,<0.3`, but it is an
direct reference pins one tag and cannot express a range, but it is an
install-time channel rather than a dependency declaration: the range is
what `pyproject.toml` carries, it is satisfied by the tag install today,
and it resolves normally once the package index exists (confirmed by the
@ -125,17 +126,38 @@ No scanning, sanitizing, or quarantine logic is implemented here.
the same floor Door B applies, with `quarantine_review` reported as its own
bucket rather than folded into rejection.
- **Upstream has released past the pin, and the decision is now taken.** `main`'s
`allow_reserved=True` kwarg shipped in guard `v0.3.3` (confirmed by a
`allow_reserved=True` kwarg was first *observed* by us in guard `v0.3.3` (a
19-fixture measurement against a scratch venv, unrelated to the pinned
install). The kwarg defaults `True`, so an unqualified call now *merges*
install). **It did not ship there** — the signature was measured across every
0.3 tag at bump time and the kwarg is present from `v0.3.0` onward, absent in
`v0.2.0`. The original wording read "first version we ran the suite against"
as "version it was introduced in"; the two coincided only because `v0.3.3`
was the first 0.3 we measured at all. The conclusion it supported was right
and the reason was wrong, so the reason is corrected rather than the outcome
quietly kept. It also decides the pin's floor: `>=0.3` is exactly right, and
would have been wrong either way if the kwarg had really arrived in `0.3.3`.
The kwarg defaults `True`, so an unqualified call now *merges*
`index.md`/`log.md` in a mode-b import instead of path-rejecting them —
reversing this plan's original "no allow_reserved toggle, rejection is
unconditional" reading. Decided: when the pin bumps into the `0.3.x` line,
`guard_adapter.import_gate` passes `allow_reserved=False` explicitly,
keeping the reserved-name refusal this plan committed to. Today the pin is
still `v0.2.0`, where the kwarg does not exist and rejection is
unconditional by construction — no code changes yet; whoever bumps the pin
adds the kwarg and a test pinning it, per assumption B1.
keeping the reserved-name refusal this plan committed to.
**Done.** The pin moved to `>=0.3,<0.4` (resolved `v0.3.4`, not `v0.3.3` —
`v0.3.4` shipped first and repairs a quadratic regex on Door C's own call
path). The 19-fixture suite was re-run against `v0.3.4` before the bump and
reproduced the `v0.3.3` deltas exactly, with none added.
`guard_adapter.import_gate` now passes `allow_reserved=False`, and
`test_door_c_pins_allow_reserved_false_against_the_guards_default` pins both
halves: that the guard still defaults `True` (without which the override is
a no-op that would pass forever over nothing) and that Door C overrides it.
The recorded justification is worth sharpening now that it is code: the
guard's `True` default is right *for the guard*, and this library does not
dispute the safety reasoning behind it. Door C's refusal is structural — it
generates the merged bundle's `index.md` from what it merged and writes every
merged concept verbatim, so a sender's `index.md` is a second and
irreconcilable claim on one path, not merely a risk to be scanned.
- **`origin`/`channel` are validated against the guard's pinned vocabulary.**
The guard derives trust from `origin` by enum *identity*, so an unrecognised
string would be silently downgraded to untrusted. The library refuses to
@ -199,5 +221,7 @@ No scanning, sanitizing, or quarantine logic is implemented here.
5. Phase 1 golden suite still passes byte-for-byte (no regression from reuse).
6. Grep-gate: `grep -rn "sanitize\|quarantine\|lexicon" src/` shows no local
security reimplementation (guard imports only).
7. `pyproject.toml` runtime dependencies == exactly `llm-ingestion-guard>=0.2,<0.3`
(automated: `test_the_only_runtime_dependency_is_the_security_boundary`).
7. `pyproject.toml` runtime dependencies == exactly one range on
`llm-ingestion-guard` (automated:
`test_the_only_runtime_dependency_is_the_security_boundary`; the range
itself is `>=0.3,<0.4` since the bump).

4
pyproject.toml
View file

@ -22,7 +22,7 @@ classifiers = [
# against a package index, and is satisfied today by the git+https tag
# install documented in the README (a direct reference is an install-time
# channel, not a dependency declaration).
dependencies = ["llm-ingestion-guard>=0.2,<0.3"]
dependencies = ["llm-ingestion-guard>=0.3,<0.4"]
[project.optional-dependencies]
# Reserved for binary file-type extraction parsers (pdf/docx/xlsx).
@ -60,4 +60,4 @@ ignore_missing_imports = true
# (README). Either way the range above stays the pin: the built wheel carries
# `Requires-Dist: llm-ingestion-guard<0.3,>=0.2`, verified against the wheel.
[tool.uv.sources]
llm-ingestion-guard = { git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git", tag = "v0.2.0" }
llm-ingestion-guard = { git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git", tag = "v0.3.4" }

20
src/llm_ingestion_okf/guard_adapter.py
View file

@ -31,6 +31,19 @@ seam is for.
cross-link graph across concepts, so gating them one at a time would throw
half the gate away. The adapter maps each `ConceptResult` to an
`ImportDecision` and returns the guard's log body unwritten.
**Door C passes `allow_reserved=False`, against the guard's default.** This is
the one place the adapter's call differs from the guard's own recommendation
for the path it is on, so it is stated rather than left to be inferred. The
guard defaults `True` for received bundles because `index.md` / `log.md` are
legitimate structural files in a conformant third-party bundle, and
over-blocking one is its own failure mode. That reasoning is about safety, and
this library does not dispute it. The override is structural: Door C GENERATES
the merged bundle's `index.md` from what it merged, and writes every merged
concept VERBATIM — so accepting a sender's `index.md` would put two
irreconcilable claims on one path, with no way to reconcile them that does not
break the verbatim invariant. Refusing it is this library's decision about its
own output, not a second opinion on the guard's scan.
"""
from __future__ import annotations
@ -86,7 +99,12 @@ def import_gate(bundle: dict[str, str], *, origin: str, channel: str) -> BundleD
code="import_provenance_invalid",
) from exc
result = guard_okf.import_bundle(dict(bundle), origin=guard_origin, channel=guard_channel)
result = guard_okf.import_bundle(
dict(bundle),
origin=guard_origin,
channel=guard_channel,
allow_reserved=False,
)
concepts = tuple(
ImportDecision(
path=str(concept.path),

2
src/llm_ingestion_okf/importer.py
View file

@ -50,7 +50,7 @@ from .profiles import DEFAULT
# The guard's non-blocking floor and its review queue, by VALUE (`Disposition`
# is a `str, Enum`, so the value is the stable thing to compare against across
# the pinned `>=0.2,<0.3` range). Pinned as constants here rather than imported
# the pinned `>=0.3,<0.4` range). Pinned as constants here rather than imported
# because the dependency is injected — deliberately restated independently of
# Door B's copy in `inbox.py`, so drift in either door is visible rather than
# silently shared. The step-4 adapter's signature smoke test is what catches a

2
src/llm_ingestion_okf/inbox.py
View file

@ -133,7 +133,7 @@ def render_inbox_concept(
# The guard's non-blocking floor. `Disposition` is a `str, Enum` in
# llm-ingestion-guard, so its VALUE is the stable thing to compare against
# across the pinned `>=0.2,<0.3` range. Pinned as a constant here rather than
# across the pinned `>=0.3,<0.4` range. Pinned as a constant here rather than
# imported, because the dependency is injected (see `Gate`): the step-4
# adapter's signature smoke test is what catches a rename in the guard.
_DISPOSITION_PERSIST = "warn"

44
tests/test_guard_adapter.py
View file

@ -22,6 +22,7 @@ from __future__ import annotations
import inspect
from pathlib import Path
from typing import Any
import pytest
@ -57,11 +58,11 @@ def _snapshot(directory: Path) -> dict[str, bytes]:
def test_guard_version_is_inside_the_pin() -> None:
"""The pin is `>=0.2,<0.3`; a 0.3 in the environment invalidates every
"""The pin is `>=0.3,<0.4`; a version outside it invalidates every
by-value comparison below and must fail loudly rather than be discovered
through a mis-branched verdict."""
major, minor = (int(part) for part in guard.__version__.split(".")[:2])
assert (major, minor) == (0, 2), guard.__version__
assert (major, minor) == (0, 3), guard.__version__
def test_guard_screen_output_signature_is_what_door_b_calls() -> None:
@ -73,11 +74,48 @@ def test_guard_screen_output_signature_is_what_door_b_calls() -> None:
def test_guard_import_bundle_signature_is_what_door_c_calls() -> None:
parameters = inspect.signature(guard_okf.import_bundle).parameters
assert list(parameters) == ["bundle", "origin", "channel"]
assert list(parameters) == ["bundle", "origin", "channel", "allow_reserved"]
# origin/channel keyword-only at the guard too: a transposed positional
# call would move a bundle between trust tiers with no type error.
assert parameters["origin"].kind is inspect.Parameter.KEYWORD_ONLY
assert parameters["channel"].kind is inspect.Parameter.KEYWORD_ONLY
assert parameters["allow_reserved"].kind is inspect.Parameter.KEYWORD_ONLY
def test_door_c_pins_allow_reserved_false_against_the_guards_default(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Door C passes `allow_reserved=False` EXPLICITLY, and that is load-bearing.
The guard defaults it `True` on the mode-b received-bundle path, reasoning
that `index.md`/`log.md` are legitimate structural files in a conformant
third-party bundle. Door C IS that path and overrides it anyway, because
this library GENERATES the bundle's `index.md` from what it merged: a
sender's `index.md`, which Door C's other invariant would write verbatim,
is a second and unreconcilable claim about the same file. The refusal is
not a security judgement layered over the guard's — it is this library's
own structural one, and it is the posture the phase-2 plan committed to
before the kwarg existed.
The first assertion is why this test cannot be dropped as redundant: the
override only means something while the guard's default disagrees with it.
Were the guard to default `False` later, the explicit kwarg would become a
no-op and this test says so, rather than passing forever over nothing.
"""
parameters = inspect.signature(guard_okf.import_bundle).parameters
assert parameters["allow_reserved"].default is True
captured: dict[str, object] = {}
real_import_bundle = guard_okf.import_bundle
def _spy(bundle: dict[str, str], **kwargs: Any) -> Any:
captured.update(kwargs)
return real_import_bundle(bundle, **kwargs)
monkeypatch.setattr(guard_adapter.guard_okf, "import_bundle", _spy)
guard_adapter.import_gate({"index.md": BENIGN}, origin="external", channel="automatic")
assert captured["allow_reserved"] is False
def test_disposition_vocabulary_matches_the_constants_the_doors_branch_on() -> None:

2
tests/test_packaging.py
View file

@ -31,7 +31,7 @@ def test_the_only_runtime_dependency_is_the_security_boundary() -> None:
"""
tomllib = pytest.importorskip("tomllib") # stdlib from 3.11; the pin holds on 3.10 too
pyproject = tomllib.loads((PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
assert pyproject["project"]["dependencies"] == ["llm-ingestion-guard>=0.2,<0.3"]
assert pyproject["project"]["dependencies"] == ["llm-ingestion-guard>=0.3,<0.4"]
def test_the_declared_version_agrees_with_the_packaged_one() -> None:

8
uv.lock generated
View file

@ -166,12 +166,12 @@ wheels = [
[[package]]
name = "llm-ingestion-guard"
version = "0.2.0"
source = { git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git?tag=v0.2.0#542ac92349cd0015972105ba66017b36cce1e708" }
version = "0.3.4"
source = { git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git?tag=v0.3.4#adf93e47fbcdbd76bdb9baf6393611230342397a" }
[[package]]
name = "llm-ingestion-okf"
version = "0.4.0"
version = "0.5.0a2"
source = { editable = "." }
dependencies = [
{ name = "llm-ingestion-guard" },
@ -185,7 +185,7 @@ dev = [
]
[package.metadata]
requires-dist = [{ name = "llm-ingestion-guard", git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git?tag=v0.2.0" }]
requires-dist = [{ name = "llm-ingestion-guard", git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git?tag=v0.3.4" }]
provides-extras = ["extract"]
[package.metadata.requires-dev]