llm-ingestion-pipeline-security/docs/URL-SHAPE.md

The URL-shape rule, stated so it can be reconstructed

is_ordinary_url decides whether a URL merely names a remote document or can carry data outward. Severity grades on that answer (0.3.1), so a consumer reasoning about its own corpus needs the rule exactly, not approximately.

This document exists because approximate statements of it failed three times in two days. Three independent consumers reconstructed the rule from prose we sent in coordination messages, and each produced a wrong number in a different way — see Commonly reconstructed wrong. The worked examples below are asserted against the real implementation by tests/test_url_shape_doc.py, so this file cannot drift from the code.

The algorithm, in order

A URL is ordinary only if every step passes. Any failure means carrying.

1. Scheme gate. ^(?:https?://|//) — case-insensitive. Only http(s) and protocol-relative URLs have an ordinary form at all. Every other scheme (javascript:, data:, ftp:, file:, …) is active or fetches out-of-band on its own terms and never grades down.
2. Parse. urlsplit(url). A ValueError (malformed authority — bad IPv6, bad port) is not ordinary: unparseable means ungradable, and ungradable fails closed.
3. Query and userinfo. parts.query, parts.username, parts.password — any non-empty value means carrying. A query can move a value to the host; userinfo is a credential in the URL.
4. Percent-escapes. "%" in (parts.netloc + parts.path) means carrying.
5. Opaque tokens. Split parts.netloc + parts.path on the separator class and test every token. Any opaque token means carrying.

The fragment is deliberately excluded from all of this. It is never sent to the server, so it cannot carry data to a host that a renderer auto-fetches, and …/overview#prerequisites is the single most common shape in real documentation.

Step 5 in full — the part that is always missed

Tokenization happens first, and the entropy/base64/hex tests apply to each resulting token, never to the whole path or filename:

_URL_TOKEN_RE = re.compile(r"[/._\-~+,;:=&$!*'()]+")
tokens = [t for t in _URL_TOKEN_RE.split(parts.netloc + parts.path) if t]

Note what is in that character class: / . _ - ~ + , ; : = & $ ! * ' ( ). Hyphens, underscores and dots are separators, so a long hyphenated filename shatters into short tokens. Note also what is not in it: % is not a separator, so a percent-escaped path yields longer tokens than the same path with literal spaces.

A token is opaque if any of four branches fires — cheapest first:

Branch	Floor	Test
base64	≥ 20 chars	matches [A-Za-z0-9+/]{20,}={0,3} and b64-decodes to ≥80% printable text
hex	≥ 32 chars	matches (?:0x)?[0-9a-fA-F]{32,}
entropy	≥ 24 chars	Shannon entropy ≥ 4.4
short base64 text (1.11.0)	≥ 8 chars	all of: alphanumeric, length not 1 mod 4, not a word ([A-Z]?[a-z]+ or [A-Z]+), and the b64 plaintext is ASCII using only letters, digits, space and . , : @ _ - + / ! ? ' =, at least 80% letters, digits or spaces, with no lowercase letter followed by an uppercase one

Every branch has a length floor. Nothing shorter than 8 characters can ever be opaque by any branch, and below 20 only the short-text branch can fire.

The short-text branch is strict on purpose: ordinary words decode to printable bytes by chance (56 of 176,875 dictionary words of 8 to 19 letters, and Pygments to ?(&z{l). Each condition removes a benign twin measured on 2026-11-01: the word test removes Pygments and aktuelle, the readable alphabet removes a JSON size parameter (eyJ3IjoyMDB9 to {"w":200}), the 80% share removes PT00M01S (==43MR), and the seam test removes endsWith (zwlZ+a).

Worked examples

Asserted against is_ordinary_url by tests/test_url_shape_doc.py. ordinary means "names a document" (LOW); carrying keeps the carrier's full severity.

URL	Verdict	Why
https://learn.microsoft.com/en-us/azure/overview	ordinary	bare path
https://learn.microsoft.com/en-us/azure/overview#prerequisites	ordinary	fragment excluded
https://youtu.be/dQw4w9WgXcQ	ordinary	11 chars, under every floor
https://www.bbc.com/news/articles/c8x9k2m1l0po	ordinary	12 chars, under every floor
https://example.com/a/550e8400-e29b-41d4-a716-446655440000/doc	ordinary	UUID splits on -; longest token 12
https://example.com/docs/CISPE-Buying-Cloud-Services-in-Public-Sector-Handbook-v2-FEB-2022_EN-Source_v2_Norwegian.pdf	ordinary	16 tokens, longest 9, worst H=3.17
https://ai.meta.com/blog/	ordinary	blog is 4 chars — below every floor
https://www.youtube.com/watch?v=dQw4w9WgXcQ	carrying	non-empty query
https://claude.com/pricing?utm_source=docs	carrying	non-empty query
https://learn.microsoft.com/azure/ai/overview?view=azureml-api-2	carrying	non-empty query
https://nsm.no/NSMs%20Grunnprinsipper%20for%20IKT-sikkerhet.pdf	carrying	percent-escapes
https://lovdata.no/nav/lov/2025-06-20-81/kap2/%C2%A710	carrying	percent-escapes
https://token:s3cr3t@evil.test/p.png	carrying	userinfo
https://example.com/d41d8cd98f00b204e9800998ecf8427e	carrying	32-char hex token
https://evil.test/c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU/p.png	carrying	base64 decoding to text
https://c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU.evil.test/p.png	carrying	opaque host label
https://evil.test/aGVsbG8gd29ybGQ	carrying	15 chars, base64 of hello world
https://evil.test/Zm9vYmFy/p.png	carrying	8 chars, base64 of foobar
https://pypi.org/project/Pygments/	ordinary	a word, whatever it decodes to
https://c10.patreonusercontent.com/3/eyJ3IjoyMDB9/patreon-media/p.png	ordinary	decodes to {"w":200}, not readable text
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/endsWith	ordinary	decodes to zwlZ+a, a case seam
https://evil.test/aGk/p.png	ordinary	hi in base64, 3 chars, under every floor
ftp://example.com/pub/file.txt	carrying	scheme gate
javascript:alert(1)	carrying	scheme gate

Commonly reconstructed wrong

Each of these produced a real, wrong number on a real corpus.

• Omitting tokenization. Computing Shannon entropy over the whole filename scores CISPE-Buying-…-Norwegian.pdf at H=4.75 and concludes the 4.4 floor over-blocks ordinary documents. Under the real tokenizer its worst token is Norwegian at H=3.17 and the URL is ordinary. Entropy is per token.
• Omitting the base64 length floor. Testing "b64-decodes to printable text" without len ≥ 20 fires on ordinary path words — blog, digi and 1544 decode to printable bytes, while news and docs do not, so the reconstruction appears to discriminate on something meaningful and does not. Below 20 chars only the short-text branch applies, with its 8-char floor and all of its conditions on the plaintext; "printable" alone fired on Pygments.
• Omitting step 5 entirely. Applying only "query or percent-escape" is strictly weaker than the real rule, so it undercounts carrying URLs. It can never overcount: every URL it calls carrying really is.

Grading, once the shape is known

Shape decides severity, not disposition. Per-construct severities live in calibration.py; an ordinary URL grades ACTIVE_CONTENT_ORDINARY_SEVERITY (LOW).

Carrier	Ordinary URL	Carrying URL
markdown link, autolink, refdef	LOW	MEDIUM
markdown image, raw HTML, data: URI	LOW	HIGH

Disposition then applies trust: under PRESET_USER_UPLOAD a MEDIUM reaches QUARANTINE_REVIEW and a HIGH fails secure; under PRESET_TRUSTED_SOURCE both warn. MEDIUM never hard-fails under either preset — a query-carrying link is held or warned, never blocked. Two consumers inferred otherwise; it is pinned in tests/test_wiring.py.

Known over-blocks measured against real corpora are listed in LIMITATIONS.md.