llm-ingestion-pipeline-security/docs/URL-SHAPE.md

# The URL-shape rule, stated so it can be reconstructed

`is_ordinary_url` decides whether a URL merely *names* a remote document or can
*carry data outward*. Severity grades on that answer (0.3.1), so a consumer
reasoning about its own corpus needs the rule exactly, not approximately.

**This document exists because approximate statements of it failed three times in
two days.** Three independent consumers reconstructed the rule from prose we sent
in coordination messages, and each produced a wrong number in a different way —
see [Commonly reconstructed wrong](#commonly-reconstructed-wrong). The worked
examples below are asserted against the real implementation by
`tests/test_url_shape_doc.py`, so this file cannot drift from the code.

## The algorithm, in order

A URL is **ordinary** only if every step passes. Any failure means *carrying*.

1. **Scheme gate.** `^(?:https?://|//)` — case-insensitive. Only http(s) and
   protocol-relative URLs have an ordinary form at all. Every other scheme
   (`javascript:`, `data:`, `ftp:`, `file:`, …) is active or fetches out-of-band on
   its own terms and never grades down.
2. **Parse.** `urlsplit(url)`. A `ValueError` (malformed authority — bad IPv6, bad
   port) is *not ordinary*: unparseable means ungradable, and ungradable fails closed.
3. **Query and userinfo.** `parts.query`, `parts.username`, `parts.password` — any
   non-empty value means carrying. A query can move a value to the host; userinfo
   is a credential in the URL.
4. **Percent-escapes.** `"%" in (parts.netloc + parts.path)` means carrying.
5. **Opaque tokens.** Split `parts.netloc + parts.path` on the separator class and
   test **every** token. Any opaque token means carrying.

The **fragment is deliberately excluded** from all of this. It is never sent to the
server, so it cannot carry data to a host that a renderer auto-fetches, and
`…/overview#prerequisites` is the single most common shape in real documentation.

### Step 5 in full — the part that is always missed

Tokenization happens **first**, and the entropy/base64/hex tests apply to each
resulting token, **never to the whole path or filename**:

```python
_URL_TOKEN_RE = re.compile(r"[/._\-~+,;:=&$!*'()]+")
tokens = [t for t in _URL_TOKEN_RE.split(parts.netloc + parts.path) if t]
```

Note what is in that character class: **`/` `.` `_` `-` `~` `+` `,` `;` `:` `=` `&`
`$` `!` `*` `'` `(` `)`**. Hyphens, underscores and dots are separators, so a long
hyphenated filename shatters into short tokens. Note also what is *not* in it:
`%` is not a separator, so a percent-escaped path yields *longer* tokens than the
same path with literal spaces.

A token is **opaque** if any of four branches fires — cheapest first:

| Branch | Floor | Test |
|---|---|---|
| base64 | **≥ 20 chars** | matches `[A-Za-z0-9+/]{20,}={0,3}` **and** b64-decodes to ≥80% printable text |
| hex | **≥ 32 chars** | matches `(?:0x)?[0-9a-fA-F]{32,}` |
| entropy | **≥ 24 chars** | Shannon entropy ≥ **4.4** |
| short base64 text (1.11.0) | **≥ 8 chars** | all of: alphanumeric, length not 1 mod 4, **not** a word (`[A-Z]?[a-z]+` or `[A-Z]+`), and the b64 plaintext is ASCII using only letters, digits, space and `. , : @ _ - + / ! ? ' =`, at least **80%** letters, digits or spaces, with **no** lowercase letter followed by an uppercase one |

Every branch has a length floor. Nothing shorter than 8 characters can ever be
opaque by any branch, and below 20 only the short-text branch can fire.

The short-text branch is strict on purpose: ordinary words decode to printable
bytes by chance (56 of 176,875 dictionary words of 8 to 19 letters, and `Pygments`
to `?(&z{l`). Each
condition removes a benign twin measured on 2026-11-01: the word test removes
`Pygments` and `aktuelle`, the readable alphabet removes a JSON size parameter
(`eyJ3IjoyMDB9` to `{"w":200}`), the 80% share removes `PT00M01S` (`==43MR`), and
the seam test removes `endsWith` (`zwlZ+a`).

## Worked examples

Asserted against `is_ordinary_url` by `tests/test_url_shape_doc.py`. `ordinary`
means "names a document" (LOW); `carrying` keeps the carrier's full severity.

| URL | Verdict | Why |
|---|---|---|
| `https://learn.microsoft.com/en-us/azure/overview` | ordinary | bare path |
| `https://learn.microsoft.com/en-us/azure/overview#prerequisites` | ordinary | fragment excluded |
| `https://youtu.be/dQw4w9WgXcQ` | ordinary | 11 chars, under every floor |
| `https://www.bbc.com/news/articles/c8x9k2m1l0po` | ordinary | 12 chars, under every floor |
| `https://example.com/a/550e8400-e29b-41d4-a716-446655440000/doc` | ordinary | UUID splits on `-`; longest token 12 |
| `https://example.com/docs/CISPE-Buying-Cloud-Services-in-Public-Sector-Handbook-v2-FEB-2022_EN-Source_v2_Norwegian.pdf` | ordinary | 16 tokens, longest 9, worst H=3.17 |
| `https://ai.meta.com/blog/` | ordinary | `blog` is 4 chars — below every floor |
| `https://www.youtube.com/watch?v=dQw4w9WgXcQ` | carrying | non-empty query |
| `https://claude.com/pricing?utm_source=docs` | carrying | non-empty query |
| `https://learn.microsoft.com/azure/ai/overview?view=azureml-api-2` | carrying | non-empty query |
| `https://nsm.no/NSMs%20Grunnprinsipper%20for%20IKT-sikkerhet.pdf` | carrying | percent-escapes |
| `https://lovdata.no/nav/lov/2025-06-20-81/kap2/%C2%A710` | carrying | percent-escapes |
| `https://token:s3cr3t@evil.test/p.png` | carrying | userinfo |
| `https://example.com/d41d8cd98f00b204e9800998ecf8427e` | carrying | 32-char hex token |
| `https://evil.test/c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU/p.png` | carrying | base64 decoding to text |
| `https://c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU.evil.test/p.png` | carrying | opaque host label |
| `https://evil.test/aGVsbG8gd29ybGQ` | carrying | 15 chars, base64 of `hello world` |
| `https://evil.test/Zm9vYmFy/p.png` | carrying | 8 chars, base64 of `foobar` |
| `https://pypi.org/project/Pygments/` | ordinary | a word, whatever it decodes to |
| `https://c10.patreonusercontent.com/3/eyJ3IjoyMDB9/patreon-media/p.png` | ordinary | decodes to `{"w":200}`, not readable text |
| `https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/endsWith` | ordinary | decodes to `zwlZ+a`, a case seam |
| `https://evil.test/aGk/p.png` | ordinary | `hi` in base64, 3 chars, under every floor |
| `ftp://example.com/pub/file.txt` | carrying | scheme gate |
| `javascript:alert(1)` | carrying | scheme gate |

## Commonly reconstructed wrong

Each of these produced a real, wrong number on a real corpus.

- **Omitting tokenization.** Computing Shannon entropy over the whole filename
  scores `CISPE-Buying-…-Norwegian.pdf` at H=4.75 and concludes the 4.4 floor
  over-blocks ordinary documents. Under the real tokenizer its worst token is
  `Norwegian` at H=3.17 and the URL is ordinary. Entropy is **per token**.
- **Omitting the base64 length floor.** Testing "b64-decodes to printable text"
  without `len ≥ 20` fires on ordinary path words — `blog`, `digi` and `1544`
  decode to printable bytes, while `news` and `docs` do not, so the reconstruction
  appears to discriminate on something meaningful and does not. Below 20 chars
  only the short-text branch applies, with its 8-char floor and all of its
  conditions on the plaintext; "printable" alone fired on `Pygments`.
- **Omitting step 5 entirely.** Applying only "query or percent-escape" is
  *strictly weaker* than the real rule, so it undercounts carrying URLs. It can
  never overcount: every URL it calls carrying really is.

## Grading, once the shape is known

Shape decides *severity*, not disposition. Per-construct severities live in
`calibration.py`; an ordinary URL grades `ACTIVE_CONTENT_ORDINARY_SEVERITY` (LOW).

| Carrier | Ordinary URL | Carrying URL |
|---|---|---|
| markdown link, autolink, refdef | LOW | MEDIUM |
| markdown image, raw HTML, `data:` URI | LOW | HIGH |

Disposition then applies trust: under `PRESET_USER_UPLOAD` a MEDIUM reaches
QUARANTINE_REVIEW and a HIGH fails secure; under `PRESET_TRUSTED_SOURCE` both warn.
**MEDIUM never hard-fails under either preset** — a query-carrying *link* is held or
warned, never blocked. Two consumers inferred otherwise; it is pinned in
`tests/test_wiring.py`.

Known over-blocks measured against real corpora are listed in
[`LIMITATIONS.md`](LIMITATIONS.md).