open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity

feat(sanitize,fence,neutralize): reject oversize input instead of half-transforming it

Browse source 
The scanners cap by truncating: they return findings, so reading a prefix costs
detection in the tail and nothing else. The three transform surfaces return
*content*, where the same move is not available — a shortened document is silent
data loss, and a transformed prefix followed by an untransformed tail is a
bypass, since the attacker chooses where in the document the payload sits.

So they fail secure instead. Above MAX_INPUT_CHARS (1 000 000) sanitize, fence
and neutralize raise OversizeInputError. sanitize is step 1 of prepare_input and
only ever removes, so that one refusal bounds the whole input path.

OversizeInputError subclasses ContractViolation: a pipeline already bracketing
its quarantined stage keeps failing closed rather than meeting a type it has
never heard of. It inherits the alert-routable property too — sizes in the
message, refusing surface in details, no input in either.

Invariant now pinned across all three: returned text is always fully
transformed, or not returned at all.

Still uncapped and recorded in LIMITATIONS: scan_active_content called directly
(through scan_output it inherits that cap) and the okf link graph. Both are
detection-shaped, so truncate-and-flag transfers unchanged — mechanical, not
policy.

699 tests (+23), coverage 128/128 + 6/6, ReDoS sweep 0 candidates / 150.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-02 21:13:08 +02:00
commit 2d98d6809d
10 changed files with 272 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

11
README.md
View file

@ -3,7 +3,7 @@
![Version](https://img.shields.io/badge/version-0.3.4-blue)
![Status](https://img.shields.io/badge/status-alpha-orange)
![Python](https://img.shields.io/badge/python-3.10%2B-purple)
![Tests](https://img.shields.io/badge/tests-676_passing-green)
![Tests](https://img.shields.io/badge/tests-699_passing-green)
![License](https://img.shields.io/badge/license-MIT-lightgrey)
**Write-time ingestion is the trust boundary that query-time guardrails
@ -90,6 +90,15 @@ the disposition is `FAIL_SECURE`, never a silent persist. Pass
together with a transform failure is treated as a probable forced-fallback attack
and halts regardless of trust tier.
`prepare_input` fails **closed** on size too: above `MAX_INPUT_CHARS`
(1 000 000) it raises `OversizeInputError`, a `ContractViolation` subclass, rather
than returning a half-sanitized document. The scanners bound their work by
reading a prefix and flagging, which costs only detection in the tail; a
transform returns *content*, where the same move would either drop your data
silently or hand back an untransformed tail — the exact place an attacker would
put the payload. Catch it where you catch your other ingest refusals; the
exception carries sizes and the refusing surface, never any of the input.
Every primitive is also exported for pipelines that compose the checklist
themselves — `sanitize`, `scan_lexicon`, `scan_entropy`, `scan_output`,
`scan_active_content`, `neutralize`, the `decide` / `guard` disposition