open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity
4 releases 13 tags
  • v0.3.4 adf93e47fb

    v0.3.4 Stable

    ktg released this 2026-08-01 18:09:29 +00:00 | 46 commits to main since this release

    Denial-of-service fix on the INPUT path. Upgrade from 0.3.3. 0.3.3 swept
    the 83 lexicon patterns arm by arm and left every other table on 0.3.2's
    hand-written rows. Generalising the sweep over all eleven regex-bearing modules
    found three more quadratic patterns — two of them on the input path, one in
    sanitize, the first thing every ingested document touches. No disposition
    changes: recall was measured case by case and nothing was lost. Earlier tags
    are not moved.

    Fixed — three quadratic patterns, two on the input path

    Same class as everything 0.3.2 and 0.3.3 fixed: a run in front of a required
    literal, so crafted input that never supplies the literal makes every start
    position rescan the tail. Each exponent is read across four doublings, not from a
    two-point ratio.

    Pattern Crafted payload Measured @ 100 000 Exponent
    sanitize._HTML_COMMENT_RE <!-- × N 20.1 s 1.96–2.14
    active_content.URL_IN_TEXT_RE <a + A × N + > 12.99 s / 14.9 s 1.87–2.22
    okf._MD_LINK_RE [ × N 7.1 s 1.99–2.05

    These are worse than the 0.3.3 findings, and the reason is a separate finding of
    its own: MAX_SCAN_CHARS is applied in scan_lexicon and scan_output only.
    sanitize, neutralize, scan_active_content and the okf link graph accept
    input of any size, so there is no cap to extrapolate to. Now documented as a
    residual in docs/LIMITATIONS.md; extending the cap into the input path changes
    the contract for existing callers and is deliberately not done in a ReDoS patch.

    Each fix is the one the pattern's own shape allows — the 0.3.3 lesson that a fix
    choice must not be copied blindly from a neighbouring table:

    • sanitize drops the regex for a str.find scan, semantically identical to
      the lazy <!--.*?--> it replaces. Excluding < from the run would lose every
      comment containing markup (<!-- <b>x</b> --> is the ordinary case); bounding
      the run would be a one-line carrier bypass of the exact construct the stripper
      exists to remove. The module's own "no catastrophic backtracking" comment was
      wrong in the same way output's was before 0.3.2, and is corrected in place.
    • URL_IN_TEXT_RE bounds its scheme run to an RFC 3986 scheme ({0,63}).
      Bounding is safe here only because this is a defanger applied inside a tag
      already flagged active:raw-html, so padding shifts where the match starts
      rather than evading detection. A lookbehind killing interior start positions
      was measured too and rejected: it drops -http://evil.com and
      .http://x.com, a one-character evasion of the defanger. Bounded, the pattern
      runs in 0.185 s at the full 1 000 000-char cap.
    • okf._MD_LINK_RE excludes [, matching active_content.MD_LINK_RE
      exactly, including the nested-label trade already documented there.

    Changed — the sweep covers every regex surface, not one table

    docs/redos-sweep.py now sweeps 150 patterns across 11 tables (0.3.3 covered
    83 in one). The collector is mechanical on both axes so no one has to remember to
    list anything: it walks each module's namespace for compiled patterns, and it
    derives each pattern's call mode from the module source, because .sub() and
    .finditer() visit every start position where .match() cannot. A pattern
    reachable only through a helper parameter gets the worst mode, marked * — the
    fallback can over-measure but never miss.

    Two arm shapes the generator cannot express are pinned by hand as a result: a tag
    that closes around a long body (repeating-unit payloads never close it), and a
    run of plain characters carrying no anchor at all. The okf destination run gets
    no row on purpose: [^)\s]+ cannot fail, so a pin for it could never go red.

    A lexicon candidate flagged at ×2.8 measured linear across four doublings
    (exponent 0.96–1.03) — the near-noise-floor false flag the script's own docstring
    warns about, confirmed a second time.

    676 tests (+10). Coverage matrix unchanged at 128/128 caught, 6/6 gaps holding.

    Downloads