1
0
Fork 0

release(0.3.4): three quadratic patterns outside the lexicon, two on the input path

0.3.3 swept 83 lexicon patterns arm by arm and left the other ten regex-bearing
modules on 0.3.2's hand-written rows. Generalising the sweep found three more,
and the two on the input path matter more than the count suggests: `sanitize` is
step 1 of `prepare_input`, and `MAX_SCAN_CHARS` is applied in `scan_lexicon` and
`scan_output` only, so there was no cap to extrapolate to. That missing input
cap is now a documented residual of its own -- extending it changes the contract
for existing callers and is not something to smuggle into a ReDoS patch.

Version synced in all four places + CHANGELOG. LIMITATIONS 30 -> 31 items;
the 0.3.3 entry claiming the lexicon sweep's scope is corrected in place, since
"the output path" was never the whole surface either.

676 tests, coverage 128/128 + 6/6 gaps, sweep clean across 150 patterns.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-01 20:08:51 +02:00
commit adf93e47fb
5 changed files with 111 additions and 18 deletions

View file

@ -7,6 +7,76 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.3.4] — 2026-08-01
> **Denial-of-service fix on the INPUT path. Upgrade from 0.3.3.** 0.3.3 swept
> the 83 lexicon patterns arm by arm and left every other table on 0.3.2's
> hand-written rows. Generalising the sweep over all eleven regex-bearing modules
> found three more quadratic patterns — two of them on the input path, one in
> `sanitize`, the first thing every ingested document touches. No disposition
> changes: recall was measured case by case and nothing was lost. Earlier tags
> are not moved.
### Fixed — three quadratic patterns, two on the input path
Same class as everything 0.3.2 and 0.3.3 fixed: a run in front of a **required**
literal, so crafted input that never supplies the literal makes every start
position rescan the tail. Each exponent is read across four doublings, not from a
two-point ratio.
| Pattern | Crafted payload | Measured @ 100 000 | Exponent |
|---|---|---|---|
| `sanitize._HTML_COMMENT_RE` | `<!--` × N | **20.1 s** | 1.962.14 |
| `active_content.URL_IN_TEXT_RE` | `<a ` + `A` × N + `>` | 12.99 s / 14.9 s | 1.872.22 |
| `okf._MD_LINK_RE` | `[` × N | 7.1 s | 1.992.05 |
These are worse than the 0.3.3 findings, and the reason is a separate finding of
its own: `MAX_SCAN_CHARS` is applied in `scan_lexicon` and `scan_output` **only**.
`sanitize`, `neutralize`, `scan_active_content` and the okf link graph accept
input of any size, so there is no cap to extrapolate to. Now documented as a
residual in `docs/LIMITATIONS.md`; extending the cap into the input path changes
the contract for existing callers and is deliberately not done in a ReDoS patch.
Each fix is the one the pattern's own shape allows — the 0.3.3 lesson that a fix
choice must not be copied blindly from a neighbouring table:
- **`sanitize`** drops the regex for a `str.find` scan, semantically identical to
the lazy `<!--.*?-->` it replaces. Excluding `<` from the run would lose every
comment containing markup (`<!-- <b>x</b> -->` is the ordinary case); bounding
the run would be a one-line carrier bypass of the exact construct the stripper
exists to remove. The module's own "no catastrophic backtracking" comment was
wrong in the same way `output`'s was before 0.3.2, and is corrected in place.
- **`URL_IN_TEXT_RE`** bounds its scheme run to an RFC 3986 scheme (`{0,63}`).
Bounding is safe *here* only because this is a defanger applied inside a tag
already flagged `active:raw-html`, so padding shifts where the match starts
rather than evading detection. A lookbehind killing interior start positions
was measured too and **rejected**: it drops `-http://evil.com` and
`.http://x.com`, a one-character evasion of the defanger. Bounded, the pattern
runs in 0.185 s at the full 1 000 000-char cap.
- **`okf._MD_LINK_RE`** excludes `[`, matching `active_content.MD_LINK_RE`
exactly, including the nested-label trade already documented there.
### Changed — the sweep covers every regex surface, not one table
`docs/redos-sweep.py` now sweeps **150 patterns across 11 tables** (0.3.3 covered
83 in one). The collector is mechanical on both axes so no one has to remember to
list anything: it walks each module's namespace for compiled patterns, and it
derives each pattern's call mode from the module source, because `.sub()` and
`.finditer()` visit every start position where `.match()` cannot. A pattern
reachable only through a helper parameter gets the worst mode, marked `*` — the
fallback can over-measure but never miss.
Two arm shapes the generator cannot express are pinned by hand as a result: a tag
that *closes* around a long body (repeating-unit payloads never close it), and a
run of plain characters carrying no anchor at all. The `okf` destination run gets
no row on purpose: `[^)\s]+` cannot fail, so a pin for it could never go red.
A lexicon candidate flagged at ×2.8 measured **linear** across four doublings
(exponent 0.961.03) — the near-noise-floor false flag the script's own docstring
warns about, confirmed a second time.
676 tests (+10). Coverage matrix unchanged at 128/128 caught, 6/6 gaps holding.
## [0.3.3] — 2026-07-31
> **Denial-of-service fix, and a correction to 0.3.2. Upgrade from 0.3.2.** The