1
0
Fork 0

release(0.3.4): three quadratic patterns outside the lexicon, two on the input path

0.3.3 swept 83 lexicon patterns arm by arm and left the other ten regex-bearing
modules on 0.3.2's hand-written rows. Generalising the sweep found three more,
and the two on the input path matter more than the count suggests: `sanitize` is
step 1 of `prepare_input`, and `MAX_SCAN_CHARS` is applied in `scan_lexicon` and
`scan_output` only, so there was no cap to extrapolate to. That missing input
cap is now a documented residual of its own -- extending it changes the contract
for existing callers and is not something to smuggle into a ReDoS patch.

Version synced in all four places + CHANGELOG. LIMITATIONS 30 -> 31 items;
the 0.3.3 entry claiming the lexicon sweep's scope is corrected in place, since
"the output path" was never the whole surface either.

676 tests, coverage 128/128 + 6/6 gaps, sweep clean across 150 patterns.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-01 20:08:51 +02:00
commit adf93e47fb
5 changed files with 111 additions and 18 deletions

View file

@ -255,6 +255,9 @@ items; this is the full list, each with the mechanism.
output path too, they were reachable through `scan_output`: `"[" * 100_000`
took 334.7s through the gate. The claim was too broad because the sweep behind
it drove `[` only through `scan_active_content`, never through the lexicon.
0.3.4 then found three more outside the lexicon — two of them on the *input*
path, where no cap applies at all — so "the output path" was never the whole
surface either.
**Measured, both claims are narrower than they read.** The new label costs no
consumer a disposition: any text containing a literal `<script>` already
produced `active:raw-html` at HIGH on 0.3.1 — so the same prose disposed
@ -280,18 +283,38 @@ items; this is the full list, each with the mechanism.
`PRESET_USER_UPLOAD` it still `fail_secure`s. The credential is never silently
missed; on one preset it is held for review instead of halted.
- **The ReDoS sweep of the lexicon has a measured sensitivity floor, not a clean
bill of health.** All 83 patterns were swept arm by arm — payloads synthesised
per run from each pattern's own skeleton, so `[`, `[system]` and `[system](`
are each probed separately rather than relying on generic units. Two patterns
were quadratic and both are fixed. But the sweep flags on *timing*, and it
ignores measurements below a 1.5 ms noise floor at N=8000. A quadratic arm
sitting just under that floor would still cost **up to ~23 s** at the
1 000 000-char cap the gate accepts. So the claim this sweep supports is
"no arm worse than ~23 s at the cap", not "no quadratic arm remains". The
method's blind spot is real and was demonstrated in this very sweep: a
generic-payload pass found only one of the two patterns, and the second
surfaced only after the payloads were generated per run.
- **The ReDoS sweep has a measured sensitivity floor, not a clean bill of
health.** All 150 compiled patterns across all eleven regex-bearing modules are
swept arm by arm — payloads synthesised per run from each pattern's own
skeleton, so `[`, `[system]` and `[system](` are each probed separately rather
than relying on generic units, and each pattern is timed in the call mode the
production code uses (`.sub()`/`.finditer()` visit every start position where
`.match()` cannot). Five patterns were quadratic across 0.3.3 and 0.3.4; all
are fixed. But the sweep flags on *timing*, and it ignores measurements below a
1.5 ms noise floor at N=8000. A quadratic arm sitting just under that floor
would still cost **up to ~23 s** at the 1 000 000-char cap. So the claim this
sweep supports is "no arm worse than ~23 s at the cap", not "no quadratic arm
remains". The method's blind spot is real and has now been demonstrated twice:
a generic-payload pass found only one of 0.3.3's two patterns, and 0.3.2's
hand-written rows missed all three of 0.3.4's — including one on `sanitize`,
the first thing every ingested document touches. **Two arm shapes the unit-
repetition payloads cannot express** are pinned by hand as a result: a tag that
*closes* around a long body, and a run of plain characters carrying no anchor
at all.
- **Only `scan_lexicon` and `scan_output` cap their input; the input-side entry
points do not.** `MAX_SCAN_CHARS` (1 000 000) is applied in those two functions
only. `sanitize`, `fence`, `neutralize`, `scan_active_content` and the okf link
graph accept text of any length, so their cost is bounded by the caller's
input, not by this library. Every *known* quadratic run on those paths is
fixed, and the residual above states what the sweep can and cannot claim — but
where an output-path residual is capped at ~23 s, the same residual on the
input path has no ceiling. A caller that ingests untrusted documents of
unbounded size should impose its own limit before `prepare_input`. Extending
the cap into the input path is deliberately **not** done as part of a ReDoS
patch: it changes the contract for existing callers (what happens to the
truncated remainder is a policy question), and that deserves its own decision
rather than being smuggled in.
## The six documented gaps (tracked by the coverage matrix)