release(0.3.4): three quadratic patterns outside the lexicon, two on the input path
0.3.3 swept 83 lexicon patterns arm by arm and left the other ten regex-bearing modules on 0.3.2's hand-written rows. Generalising the sweep found three more, and the two on the input path matter more than the count suggests: `sanitize` is step 1 of `prepare_input`, and `MAX_SCAN_CHARS` is applied in `scan_lexicon` and `scan_output` only, so there was no cap to extrapolate to. That missing input cap is now a documented residual of its own -- extending it changes the contract for existing callers and is not something to smuggle into a ReDoS patch. Version synced in all four places + CHANGELOG. LIMITATIONS 30 -> 31 items; the 0.3.3 entry claiming the lexicon sweep's scope is corrected in place, since "the output path" was never the whole surface either. 676 tests, coverage 128/128 + 6/6 gaps, sweep clean across 150 patterns.
This commit is contained in:
parent
73fa1b99ae
commit
adf93e47fb
5 changed files with 111 additions and 18 deletions
70
CHANGELOG.md
70
CHANGELOG.md
|
|
@ -7,6 +7,76 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.3.4] — 2026-08-01
|
||||
|
||||
> **Denial-of-service fix on the INPUT path. Upgrade from 0.3.3.** 0.3.3 swept
|
||||
> the 83 lexicon patterns arm by arm and left every other table on 0.3.2's
|
||||
> hand-written rows. Generalising the sweep over all eleven regex-bearing modules
|
||||
> found three more quadratic patterns — two of them on the input path, one in
|
||||
> `sanitize`, the first thing every ingested document touches. No disposition
|
||||
> changes: recall was measured case by case and nothing was lost. Earlier tags
|
||||
> are not moved.
|
||||
|
||||
### Fixed — three quadratic patterns, two on the input path
|
||||
|
||||
Same class as everything 0.3.2 and 0.3.3 fixed: a run in front of a **required**
|
||||
literal, so crafted input that never supplies the literal makes every start
|
||||
position rescan the tail. Each exponent is read across four doublings, not from a
|
||||
two-point ratio.
|
||||
|
||||
| Pattern | Crafted payload | Measured @ 100 000 | Exponent |
|
||||
|---|---|---|---|
|
||||
| `sanitize._HTML_COMMENT_RE` | `<!--` × N | **20.1 s** | 1.96–2.14 |
|
||||
| `active_content.URL_IN_TEXT_RE` | `<a ` + `A` × N + `>` | 12.99 s / 14.9 s | 1.87–2.22 |
|
||||
| `okf._MD_LINK_RE` | `[` × N | 7.1 s | 1.99–2.05 |
|
||||
|
||||
These are worse than the 0.3.3 findings, and the reason is a separate finding of
|
||||
its own: `MAX_SCAN_CHARS` is applied in `scan_lexicon` and `scan_output` **only**.
|
||||
`sanitize`, `neutralize`, `scan_active_content` and the okf link graph accept
|
||||
input of any size, so there is no cap to extrapolate to. Now documented as a
|
||||
residual in `docs/LIMITATIONS.md`; extending the cap into the input path changes
|
||||
the contract for existing callers and is deliberately not done in a ReDoS patch.
|
||||
|
||||
Each fix is the one the pattern's own shape allows — the 0.3.3 lesson that a fix
|
||||
choice must not be copied blindly from a neighbouring table:
|
||||
|
||||
- **`sanitize`** drops the regex for a `str.find` scan, semantically identical to
|
||||
the lazy `<!--.*?-->` it replaces. Excluding `<` from the run would lose every
|
||||
comment containing markup (`<!-- <b>x</b> -->` is the ordinary case); bounding
|
||||
the run would be a one-line carrier bypass of the exact construct the stripper
|
||||
exists to remove. The module's own "no catastrophic backtracking" comment was
|
||||
wrong in the same way `output`'s was before 0.3.2, and is corrected in place.
|
||||
- **`URL_IN_TEXT_RE`** bounds its scheme run to an RFC 3986 scheme (`{0,63}`).
|
||||
Bounding is safe *here* only because this is a defanger applied inside a tag
|
||||
already flagged `active:raw-html`, so padding shifts where the match starts
|
||||
rather than evading detection. A lookbehind killing interior start positions
|
||||
was measured too and **rejected**: it drops `-http://evil.com` and
|
||||
`.http://x.com`, a one-character evasion of the defanger. Bounded, the pattern
|
||||
runs in 0.185 s at the full 1 000 000-char cap.
|
||||
- **`okf._MD_LINK_RE`** excludes `[`, matching `active_content.MD_LINK_RE`
|
||||
exactly, including the nested-label trade already documented there.
|
||||
|
||||
### Changed — the sweep covers every regex surface, not one table
|
||||
|
||||
`docs/redos-sweep.py` now sweeps **150 patterns across 11 tables** (0.3.3 covered
|
||||
83 in one). The collector is mechanical on both axes so no one has to remember to
|
||||
list anything: it walks each module's namespace for compiled patterns, and it
|
||||
derives each pattern's call mode from the module source, because `.sub()` and
|
||||
`.finditer()` visit every start position where `.match()` cannot. A pattern
|
||||
reachable only through a helper parameter gets the worst mode, marked `*` — the
|
||||
fallback can over-measure but never miss.
|
||||
|
||||
Two arm shapes the generator cannot express are pinned by hand as a result: a tag
|
||||
that *closes* around a long body (repeating-unit payloads never close it), and a
|
||||
run of plain characters carrying no anchor at all. The `okf` destination run gets
|
||||
no row on purpose: `[^)\s]+` cannot fail, so a pin for it could never go red.
|
||||
|
||||
A lexicon candidate flagged at ×2.8 measured **linear** across four doublings
|
||||
(exponent 0.96–1.03) — the near-noise-floor false flag the script's own docstring
|
||||
warns about, confirmed a second time.
|
||||
|
||||
676 tests (+10). Coverage matrix unchanged at 128/128 caught, 6/6 gaps holding.
|
||||
|
||||
## [0.3.3] — 2026-07-31
|
||||
|
||||
> **Denial-of-service fix, and a correction to 0.3.2. Upgrade from 0.3.2.** The
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue