docs(redos): the sweep now measures on the clock its own numbers justify
docs/redos-sweep.py timed on time.monotonic() while every ReDoS bound in the suite had moved to process CPU time, so the 1.5 ms sensitivity floor and the "~23 s at the cap" figure in docs/LIMITATIONS.md were produced by a different instrument than the bounds they support. The script imports scan_seconds now. Re-derived on that instrument, the floor came back UNCHANGED at 1.5 ms. Twelve full runs of all 2585 arms: median ratio 1.95-2.03 in every size bucket above 50 us, but two-point excursions past the 2.6 threshold survive at every magnitude (p99 2.9-3.3 even above 1 ms). Flagged arms per run by floor: 6.9 at 0.5 ms, 1.1 at 1.0 ms, 0.33 at 1.5 ms. Four arms flagged across the twelve runs, each in exactly one; six arms that have ever flagged re-measure at exponent 0.97-1.09 over six doublings, at most 1.2 s at the cap. Descheduling was never what made this sweep noisy -- a ratio from two points is. Measured before publishing and it cost a correction: an earlier draft of this change said the sweep "reports 0 candidates". The next run reported 2. Nine of twelve instrumented runs are clean and eight consecutive shipped-script runs under load flagged 0-3, so neither a clean run nor a flagged one is evidence on its own -- the doc says that now. Also corrects the pattern count in the same bullet: 150 -> 152, the script's own printed total. The 0.3.4 CHANGELOG entry keeps its 150 as a snapshot. No exported surface, no detector behaviour, no calibration touched. 802 passed, 129/129 classes, 6/6 gaps, 43 limitations, gitleaks clean.
This commit is contained in:
parent
4472f209a4
commit
74656123f9
3 changed files with 90 additions and 26 deletions
21
CHANGELOG.md
21
CHANGELOG.md
|
|
@ -7,7 +7,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
Nothing yet.
|
### Changed — the ReDoS sweep now measures on the same clock as the bounds it justifies
|
||||||
|
|
||||||
|
`docs/redos-sweep.py` timed on `time.monotonic()` while every ReDoS bound in the
|
||||||
|
suite moved to process CPU time (`tests/redos_clock.py`), so the 1.5 ms
|
||||||
|
sensitivity floor and the "~23 s at the cap" figure published in
|
||||||
|
`docs/LIMITATIONS.md` came from a different instrument than the bounds they
|
||||||
|
support. The script now imports `scan_seconds` rather than timing itself.
|
||||||
|
|
||||||
|
The floor was re-derived on that instrument and **stayed at 1.5 ms**: over twelve
|
||||||
|
full runs of all 2585 arms the median ratio is 1.95-2.03 in every size bucket
|
||||||
|
above 50 µs, but two-point excursions past the 2.6 flag threshold persist at every
|
||||||
|
magnitude (p99 ratio 2.9-3.3 even above 1 ms) — 6.9 flagged arms per run at a
|
||||||
|
0.5 ms floor, 1.1 at 1.0 ms, 0.33 at 1.5 ms. Descheduling was never what made this
|
||||||
|
sweep noisy; a ratio computed from two points is. Four arms flagged across those
|
||||||
|
twelve runs, each in exactly one of them, and six arms that have ever flagged
|
||||||
|
re-measure at exponent 0.97-1.09 over six doublings — at most 1.2 s at the
|
||||||
|
1 000 000-char cap. The pattern count the script prints is 152, not the 150 of the
|
||||||
|
0.3.4 entry below; `docs/LIMITATIONS.md` now carries the current number.
|
||||||
|
|
||||||
|
No exported surface, no detector behaviour and no calibration changed.
|
||||||
|
|
||||||
|
|
||||||
## [1.1.0] — 2026-08-13
|
## [1.1.0] — 2026-08-13
|
||||||
|
|
|
||||||
|
|
@ -524,18 +524,22 @@ items; this is the full list, each with the mechanism.
|
||||||
missed; on one preset it is held for review instead of halted.
|
missed; on one preset it is held for review instead of halted.
|
||||||
|
|
||||||
- **The ReDoS sweep has a measured sensitivity floor, not a clean bill of
|
- **The ReDoS sweep has a measured sensitivity floor, not a clean bill of
|
||||||
health.** All 150 compiled patterns across all eleven regex-bearing modules are
|
health.** All 152 compiled patterns across all eleven regex-bearing modules are
|
||||||
swept arm by arm — payloads synthesised per run from each pattern's own
|
swept arm by arm — payloads synthesised per run from each pattern's own
|
||||||
skeleton, so `[`, `[system]` and `[system](` are each probed separately rather
|
skeleton, so `[`, `[system]` and `[system](` are each probed separately rather
|
||||||
than relying on generic units, and each pattern is timed in the call mode the
|
than relying on generic units, and each pattern is timed in the call mode the
|
||||||
production code uses (`.sub()`/`.finditer()` visit every start position where
|
production code uses (`.sub()`/`.finditer()` visit every start position where
|
||||||
`.match()` cannot). Five patterns were quadratic across 0.3.3 and 0.3.4; all
|
`.match()` cannot). Five patterns were quadratic across 0.3.3 and 0.3.4; all
|
||||||
are fixed. But the sweep flags on *timing*, and it ignores measurements below a
|
are fixed. But the sweep flags on *timing*, and it ignores measurements below a
|
||||||
1.5 ms noise floor at N=8000. A quadratic arm sitting just under that floor
|
1.5 ms noise floor at N=8000 — process CPU time, re-derived on that instrument
|
||||||
would still cost **up to ~23 s** at the 1 000 000-char cap. So the claim this
|
(see the clock bullet below) rather than inherited from the wall clock the
|
||||||
sweep supports is "no arm worse than ~23 s at the cap", not "no quadratic arm
|
script used through 1.1.0. A quadratic arm sitting just under that floor would
|
||||||
remains". The method's blind spot is real and has now been demonstrated twice:
|
still cost **up to ~23 s** at the 1 000 000-char cap — that figure is arithmetic
|
||||||
a generic-payload pass found only one of 0.3.3's two patterns, and 0.3.2's
|
and not a measurement: a quadratic arm costs the square of the length ratio, and
|
||||||
|
1.5 ms × 125 × 125 is 23.4 s. So the claim this sweep supports is "no arm worse
|
||||||
|
than ~23 s at the cap", not "no quadratic arm remains". The method's blind spot
|
||||||
|
is real and has now been demonstrated twice: a generic-payload pass found only
|
||||||
|
one of 0.3.3's two patterns, and 0.3.2's
|
||||||
hand-written rows missed all three of 0.3.4's — including one on `sanitize`,
|
hand-written rows missed all three of 0.3.4's — including one on `sanitize`,
|
||||||
the first thing every ingested document touches. **Two arm shapes the unit-
|
the first thing every ingested document touches. **Two arm shapes the unit-
|
||||||
repetition payloads cannot express** are pinned by hand as a result: a tag that
|
repetition payloads cannot express** are pinned by hand as a result: a tag that
|
||||||
|
|
@ -574,8 +578,8 @@ items; this is the full list, each with the mechanism.
|
||||||
the suite cannot report its own blind spot, and every bound in it should be read as
|
the suite cannot report its own blind spot, and every bound in it should be read as
|
||||||
"verified red under the vulnerable form" only where a comment says it was.
|
"verified red under the vulnerable form" only where a comment says it was.
|
||||||
|
|
||||||
- **Every ReDoS bound in the suite is measured on process CPU time; the sweep that
|
- **Every ReDoS bound in the suite is measured on process CPU time, and so is the
|
||||||
sets the published sensitivity floor is not.** `tests/redos_clock.py` is the one
|
sweep that sets the published sensitivity floor.** `tests/redos_clock.py` is the one
|
||||||
clock all six test files import — `time.process_time()` — because a blowup is spent
|
clock all six test files import — `time.process_time()` — because a blowup is spent
|
||||||
cycles while a loaded machine steals wall clock without adding any. On
|
cycles while a loaded machine steals wall clock without adding any. On
|
||||||
`time.monotonic()` two 0.7.0 rows failed at **2.24s / 3.66s** against a 2.0s bound
|
`time.monotonic()` two 0.7.0 rows failed at **2.24s / 3.66s** against a 2.0s bound
|
||||||
|
|
@ -589,12 +593,26 @@ items; this is the full list, each with the mechanism.
|
||||||
in-memory string, with no I/O and no locks — the last wall-clock holdout was retired
|
in-memory string, with no I/O and no locks — the last wall-clock holdout was retired
|
||||||
by auditing its path for anything that could block, not by assumption, and a wall
|
by auditing its path for anything that could block, not by assumption, and a wall
|
||||||
clock guarding a mode that cannot occur still charges the false-red premium
|
clock guarding a mode that cannot occur still charges the false-red premium
|
||||||
(measured there at 21.6s against a 10.0s bound, on a scan that spent 7.6s). **The
|
(measured there at 21.6s against a 10.0s bound, on a scan that spent 7.6s). **The sweep
|
||||||
divergence to know about:** `docs/redos-sweep.py` still times on `time.monotonic()`
|
now runs on the same clock, and closing that divergence bought no sensitivity.**
|
||||||
(`redos-sweep.py:139`), so the 1.5 ms floor at N=8000 and the "~23 s at the cap"
|
`docs/redos-sweep.py` imports `scan_seconds` instead of timing on
|
||||||
figure above are wall-clock numbers, produced by a different instrument than the
|
`time.monotonic()`, so the 1.5 ms floor at N=8000 and the "~23 s at the cap"
|
||||||
bounds they justify. They have not been re-derived on the CPU clock; treat them as
|
figure above are finally in the same currency as the bounds they justify. What
|
||||||
the noisier of the two measurements rather than as directly comparable.
|
the move did *not* do is quiet the sweep, and the floor came back unchanged.
|
||||||
|
Measured over **twelve full runs of all 2585 arms**: the median ratio sits at
|
||||||
|
**1.95–2.03** in every size bucket above 50 µs — the whole surface measures
|
||||||
|
linear — while two-point excursions past the 2.6 flag threshold survive at every
|
||||||
|
magnitude, p99 ratio **2.9–3.3 even above 1 ms**. Flagged arms per run by floor:
|
||||||
|
**6.9 at 0.5 ms, 1.1 at 1.0 ms, 0.33 at 1.5 ms** (0–2 per run), so 1.5 ms is
|
||||||
|
still the knee. Descheduling was never what made this sweep noisy — a ratio
|
||||||
|
computed from two points is. Four distinct arms flagged at the shipped floor
|
||||||
|
across those twelve runs, **each in exactly one of them**, and nine of the twelve
|
||||||
|
runs were clean; eight consecutive runs of the shipped script immediately after a
|
||||||
|
full test run flagged 0–3 arms each, so machine load still moves the count even
|
||||||
|
on a CPU clock. Six flagged arms re-measured over six doublings give exponent
|
||||||
|
**0.97–1.09** and at most 1.2 s at the 1 000 000-char cap. **A single clean run
|
||||||
|
of this sweep is therefore not evidence either** — and neither is a single
|
||||||
|
flagged one.
|
||||||
|
|
||||||
- **Every surface now bounds its input, but not all of them the same way.**
|
- **Every surface now bounds its input, but not all of them the same way.**
|
||||||
`sanitize`, `fence` and `neutralize` raise `OversizeInputError` above
|
`sanitize`, `fence` and `neutralize` raise `OversizeInputError` above
|
||||||
|
|
|
||||||
|
|
@ -37,17 +37,30 @@ import importlib
|
||||||
import json
|
import json
|
||||||
import re
|
import re
|
||||||
import sys
|
import sys
|
||||||
import time
|
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
SRC = Path(__file__).resolve().parent.parent / "src" / "llm_ingestion_guard"
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
SRC = ROOT / "src" / "llm_ingestion_guard"
|
||||||
sys.path.insert(0, str(SRC.parent))
|
sys.path.insert(0, str(SRC.parent))
|
||||||
|
sys.path.insert(0, str(ROOT / "tests"))
|
||||||
|
|
||||||
from llm_ingestion_guard.lexicon import load_lexicon # noqa: E402
|
from llm_ingestion_guard.lexicon import load_lexicon # noqa: E402
|
||||||
|
from redos_clock import scan_seconds # noqa: E402
|
||||||
|
|
||||||
N1, N2 = 4_000, 8_000
|
N1, N2 = 4_000, 8_000
|
||||||
RATIO_FLAG = 2.6
|
RATIO_FLAG = 2.6
|
||||||
|
# RE-DERIVED on the CPU clock, not inherited from the wall clock this script used
|
||||||
|
# through 1.1.0. The clock move fixed false REDS in the suite's bounds; it bought
|
||||||
|
# this sweep no sensitivity. Over twelve full runs (2585 arms each) the median
|
||||||
|
# ratio is 1.95-2.03 in every size bucket above 50 us -- the whole surface
|
||||||
|
# measures linear -- yet two-point excursions past RATIO_FLAG survive at every
|
||||||
|
# magnitude (p99 ratio 2.9-3.3 even above 1 ms). Flagged arms per run by floor:
|
||||||
|
# 6.9 at 0.5 ms, 1.1 at 1.0 ms, 0.33 at 1.5 ms. The knee is here. Four arms
|
||||||
|
# flagged across those twelve runs, each in exactly ONE of them, and six arms
|
||||||
|
# that have ever flagged re-measure at exponent 0.97-1.09 over six doublings.
|
||||||
|
# Descheduling was never what made this sweep noisy -- a two-point ratio is, so
|
||||||
|
# read a clean run and a flagged run with the same suspicion.
|
||||||
NOISE_FLOOR = 0.0015
|
NOISE_FLOOR = 0.0015
|
||||||
HARD_CAP = 20.0
|
HARD_CAP = 20.0
|
||||||
|
|
||||||
|
|
@ -134,21 +147,35 @@ def build(unit: str, n: int) -> str:
|
||||||
|
|
||||||
|
|
||||||
def t(rx: re.Pattern[str], text: str, mode: str = "search") -> float:
|
def t(rx: re.Pattern[str], text: str, mode: str = "search") -> float:
|
||||||
"""Time one scan of ``text`` in the mode the production code actually uses."""
|
"""Time one scan of ``text`` in the mode the production code actually uses.
|
||||||
|
|
||||||
|
On the SAME clock every ReDoS bound in the suite is measured against --
|
||||||
|
``tests/redos_clock.py``, process CPU time -- imported rather than restated
|
||||||
|
here, for the reason that module gives: a blowup is spent cycles, and a
|
||||||
|
loaded machine steals wall clock without adding any. Until 1.1.0 this timed
|
||||||
|
on ``time.monotonic()``, which made the floor below and the cap figure
|
||||||
|
derived from it numbers from a different instrument than the bounds they
|
||||||
|
justify. The closure is built BEFORE the clock starts, so only the scan is
|
||||||
|
charged.
|
||||||
|
"""
|
||||||
mode = mode.rstrip("*")
|
mode = mode.rstrip("*")
|
||||||
start = time.monotonic()
|
|
||||||
if mode == "finditer":
|
if mode == "finditer":
|
||||||
for _ in rx.finditer(text):
|
def scan(s: str) -> None:
|
||||||
pass
|
for _ in rx.finditer(s):
|
||||||
|
pass
|
||||||
elif mode == "sub":
|
elif mode == "sub":
|
||||||
rx.sub("", text)
|
def scan(s: str) -> None:
|
||||||
|
rx.sub("", s)
|
||||||
elif mode == "match":
|
elif mode == "match":
|
||||||
rx.match(text)
|
def scan(s: str) -> None:
|
||||||
|
rx.match(s)
|
||||||
elif mode == "fullmatch":
|
elif mode == "fullmatch":
|
||||||
rx.fullmatch(text)
|
def scan(s: str) -> None:
|
||||||
|
rx.fullmatch(s)
|
||||||
else:
|
else:
|
||||||
rx.search(text)
|
def scan(s: str) -> None:
|
||||||
return time.monotonic() - start
|
rx.search(s)
|
||||||
|
return scan_seconds(scan, text)
|
||||||
|
|
||||||
|
|
||||||
# --- targets ----------------------------------------------------------------
|
# --- targets ----------------------------------------------------------------
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue