1
0
Fork 0

feat(sanitize,fence,neutralize): reject oversize input instead of half-transforming it

The scanners cap by truncating: they return findings, so reading a prefix costs
detection in the tail and nothing else. The three transform surfaces return
*content*, where the same move is not available — a shortened document is silent
data loss, and a transformed prefix followed by an untransformed tail is a
bypass, since the attacker chooses where in the document the payload sits.

So they fail secure instead. Above MAX_INPUT_CHARS (1 000 000) sanitize, fence
and neutralize raise OversizeInputError. sanitize is step 1 of prepare_input and
only ever removes, so that one refusal bounds the whole input path.

OversizeInputError subclasses ContractViolation: a pipeline already bracketing
its quarantined stage keeps failing closed rather than meeting a type it has
never heard of. It inherits the alert-routable property too — sizes in the
message, refusing surface in details, no input in either.

Invariant now pinned across all three: returned text is always fully
transformed, or not returned at all.

Still uncapped and recorded in LIMITATIONS: scan_active_content called directly
(through scan_output it inherits that cap) and the okf link graph. Both are
detection-shaped, so truncate-and-flag transfers unchanged — mechanical, not
policy.

699 tests (+23), coverage 128/128 + 6/6, ReDoS sweep 0 candidates / 150.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-02 21:13:08 +02:00
commit 2d98d6809d
10 changed files with 272 additions and 21 deletions

View file

@ -302,19 +302,20 @@ items; this is the full list, each with the mechanism.
*closes* around a long body, and a run of plain characters carrying no anchor
at all.
- **Only `scan_lexicon` and `scan_output` cap their input; the input-side entry
points do not.** `MAX_SCAN_CHARS` (1 000 000) is applied in those two functions
only. `sanitize`, `fence`, `neutralize`, `scan_active_content` and the okf link
graph accept text of any length, so their cost is bounded by the caller's
input, not by this library. Every *known* quadratic run on those paths is
fixed, and the residual above states what the sweep can and cannot claim — but
where an output-path residual is capped at ~23 s, the same residual on the
input path has no ceiling. A caller that ingests untrusted documents of
unbounded size should impose its own limit before `prepare_input`. Extending
the cap into the input path is deliberately **not** done as part of a ReDoS
patch: it changes the contract for existing callers (what happens to the
truncated remainder is a policy question), and that deserves its own decision
rather than being smuggled in.
- **Two detection surfaces still accept unbounded input; the transform surfaces
no longer do.** Since 0.3.5 `sanitize`, `fence` and `neutralize` raise
`OversizeInputError` above `MAX_INPUT_CHARS` (1 000 000) rather than returning
a partially transformed document, which bounds the whole input path — `sanitize`
is step 1 of `prepare_input`, and it only ever removes, so everything after it
is already under the cap. They reject rather than truncate because they return
*content*: a shortened document is silent data loss, and a transformed prefix
followed by an untransformed tail is a bypass an attacker positions the payload
into. The scanners keep truncating, which costs only detection in the tail.
What remains uncapped is `scan_active_content` **called directly** (reached
through `scan_output` it inherits that cap) and the okf link graph, whose cost
is a bundle-wide `findall` over every document body. Both are detection-shaped,
so the scanners' truncate-and-flag mechanism transfers to them unchanged — that
is a mechanical follow-up, not a policy question, and it is not yet done.
## The six documented gaps (tracked by the coverage matrix)