open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 6 Activity

feat(active): grade a short URL token that base64-decodes to text as carrying

Browse source 
Gap 6 closed. _is_opaque has a fourth branch for host labels and path
segments of 8-19 chars (calibration.URL_SHORT_B64_MIN_LEN). It fires when
the token is alphanumeric, not 1 mod 4, not itself a word, and its base64
plaintext reads as text: only letters, digits, space and . , : @ _ - + / !
? ' =, at least 80% letters/digits/spaces (URL_SHORT_B64_TEXT_SHARE), and
no lowercase-to-uppercase seam. ![x](https://evil.test/aGVsbG8gd29ybGQ)
is now HIGH; a link with it MEDIUM.

"Printable" alone fired on Pygments, a Norwegian path word and a JSON size
parameter. Each condition removes a measured twin (Pygments, aktuelle,
Slovenly, {"w":200}, PT00M01S, endsWith). Measured before building, with
the shipped tokenizer: 0 of 22,725 URLs in the 3,583 clean documents and
577 package READMEs, 0 of 530,625 dictionary words, 9 of 336,122
identifiers and prose tokens (all real base64 of text). Recall 32 of 35.

Measured before and after on the same trees: all 4,160 documents are
byte-identical under both presets (disposition, labels, severity, count,
offset); OKF import stays 47/31; positive control WARN -> FAIL_SECURE.
is_ordinary_url about 4 us slower per URL; 1M-char worst case unchanged.

26 new tests (attacks red first), 11 of 11 mutations red. Matrix 147/147 +
4/4 gaps, suite 1300 passed, ReDoS sweep 173 patterns, 0 flags on the new.
This commit is contained in:
Kjell Tore Guttormsen 2026-11-01 20:23:23 +01:00
commit 7430d7855a
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
14 changed files with 350 additions and 50 deletions
Download patch file Download diff file Expand all files Collapse all files

6
tests/test_docs_measurement_scripts.py
View file

@ -272,13 +272,13 @@ def test_redos_sweep_floor_and_flag_match_the_published_numbers():
assert "2.6 flag threshold" in text
def test_redos_sweep_collector_covers_169_patterns_across_11_tables():
# The count docs/LIMITATIONS.md carries as "all 169 compiled patterns across
def test_redos_sweep_collector_covers_173_patterns_across_11_tables():
# The count docs/LIMITATIONS.md carries as "all 173 compiled patterns across
# all eleven regex-bearing modules". A pattern added or removed in `src/`
# without re-measuring would drift the doc's claim silently otherwise.
assert len(redos_sweep.TABLES) == 11
total = sum(len(collect()) for collect in redos_sweep.TABLES.values())
assert total == 169
assert total == 173
def test_redos_sweep_collector_walks_a_dict_of_patterns():