# The URL-shape rule, stated so it can be reconstructed

`is_ordinary_url` decides whether a URL merely *names* a remote document or can
*carry data outward*. Severity grades on that answer (0.3.1), so a consumer
reasoning about its own corpus needs the rule exactly, not approximately.

**This document exists because approximate statements of it failed three times in
two days.** Three independent consumers reconstructed the rule from prose we sent
in coordination messages, and each produced a wrong number in a different way —
see [Commonly reconstructed wrong](#commonly-reconstructed-wrong). The worked
examples below are asserted against the real implementation by
`tests/test_url_shape_doc.py`, so this file cannot drift from the code.

## The algorithm, in order

A URL is **ordinary** only if every step passes. Any failure means *carrying*.

1. **Scheme gate.** `^(?:https?://|//)` — case-insensitive. Only http(s) and
   protocol-relative URLs have an ordinary form at all. Every other scheme
   (`javascript:`, `data:`, `ftp:`, `file:`, …) is active or fetches out-of-band on
   its own terms and never grades down.
2. **Parse.** `urlsplit(url)`. A `ValueError` (malformed authority — bad IPv6, bad
   port) is *not ordinary*: unparseable means ungradable, and ungradable fails closed.
3. **Query and userinfo.** `parts.query`, `parts.username`, `parts.password` — any
   non-empty value means carrying. A query can move a value to the host; userinfo
   is a credential in the URL.
4. **Percent-escapes.** `"%" in (parts.netloc + parts.path)` means carrying.
5. **Opaque tokens.** Split `parts.netloc + parts.path` on the separator class and
   test **every** token. Any opaque token means carrying.

The **fragment is deliberately excluded** from all of this. It is never sent to the
server, so it cannot carry data to a host that a renderer auto-fetches, and
`…/overview#prerequisites` is the single most common shape in real documentation.

### Step 5 in full — the part that is always missed

Tokenization happens **first**, and the entropy/base64/hex tests apply to each
resulting token, **never to the whole path or filename**:

```python
_URL_TOKEN_RE = re.compile(r"[/._\-~+,;:=&$!*'()]+")
tokens = [t for t in _URL_TOKEN_RE.split(parts.netloc + parts.path) if t]
```

Note what is in that character class: **`/` `.` `_` `-` `~` `+` `,` `;` `:` `=` `&`
`$` `!` `*` `'` `(` `)`**. Hyphens, underscores and dots are separators, so a long
hyphenated filename shatters into short tokens. Note also what is *not* in it:
`%` is not a separator, so a percent-escaped path yields *longer* tokens than the
same path with literal spaces.

A token is **opaque** if any of three branches fires — cheapest first:

| Branch | Floor | Test |
|---|---|---|
| base64 | **≥ 20 chars** | matches `[A-Za-z0-9+/]{20,}={0,3}` **and** b64-decodes to ≥80% printable text |
| hex | **≥ 32 chars** | matches `(?:0x)?[0-9a-fA-F]{32,}` |
| entropy | **≥ 24 chars** | Shannon entropy ≥ **4.4** |

Every branch has a length floor. Nothing shorter than 20 characters can ever be
opaque by any branch.

## Worked examples

Asserted against `is_ordinary_url` by `tests/test_url_shape_doc.py`. `ordinary`
means "names a document" (LOW); `carrying` keeps the carrier's full severity.

| URL | Verdict | Why |
|---|---|---|
| `https://learn.microsoft.com/en-us/azure/overview` | ordinary | bare path |
| `https://learn.microsoft.com/en-us/azure/overview#prerequisites` | ordinary | fragment excluded |
| `https://youtu.be/dQw4w9WgXcQ` | ordinary | 11 chars, under every floor |
| `https://www.bbc.com/news/articles/c8x9k2m1l0po` | ordinary | 12 chars, under every floor |
| `https://example.com/a/550e8400-e29b-41d4-a716-446655440000/doc` | ordinary | UUID splits on `-`; longest token 12 |
| `https://example.com/docs/CISPE-Buying-Cloud-Services-in-Public-Sector-Handbook-v2-FEB-2022_EN-Source_v2_Norwegian.pdf` | ordinary | 16 tokens, longest 9, worst H=3.17 |
| `https://ai.meta.com/blog/` | ordinary | `blog` is 4 chars — below the base64 floor |
| `https://www.youtube.com/watch?v=dQw4w9WgXcQ` | carrying | non-empty query |
| `https://claude.com/pricing?utm_source=docs` | carrying | non-empty query |
| `https://learn.microsoft.com/azure/ai/overview?view=azureml-api-2` | carrying | non-empty query |
| `https://nsm.no/NSMs%20Grunnprinsipper%20for%20IKT-sikkerhet.pdf` | carrying | percent-escapes |
| `https://lovdata.no/nav/lov/2025-06-20-81/kap2/%C2%A710` | carrying | percent-escapes |
| `https://token:s3cr3t@evil.test/p.png` | carrying | userinfo |
| `https://example.com/d41d8cd98f00b204e9800998ecf8427e` | carrying | 32-char hex token |
| `https://evil.test/c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU/p.png` | carrying | base64 decoding to text |
| `https://c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU.evil.test/p.png` | carrying | opaque host label |
| `ftp://example.com/pub/file.txt` | carrying | scheme gate |
| `javascript:alert(1)` | carrying | scheme gate |

## Commonly reconstructed wrong

Each of these produced a real, wrong number on a real corpus.

- **Omitting tokenization.** Computing Shannon entropy over the whole filename
  scores `CISPE-Buying-…-Norwegian.pdf` at H=4.75 and concludes the 4.4 floor
  over-blocks ordinary documents. Under the real tokenizer its worst token is
  `Norwegian` at H=3.17 and the URL is ordinary. Entropy is **per token**.
- **Omitting the base64 length floor.** Testing "b64-decodes to printable text"
  without `len ≥ 20` fires on ordinary path words — `blog`, `digi` and `1544`
  decode to printable bytes, while `news` and `docs` do not, so the reconstruction
  appears to discriminate on something meaningful and does not.
- **Omitting step 5 entirely.** Applying only "query or percent-escape" is
  *strictly weaker* than the real rule, so it undercounts carrying URLs. It can
  never overcount: every URL it calls carrying really is.

## Grading, once the shape is known

Shape decides *severity*, not disposition. Per-construct severities live in
`calibration.py`; an ordinary URL grades `ACTIVE_CONTENT_ORDINARY_SEVERITY` (LOW).

| Carrier | Ordinary URL | Carrying URL |
|---|---|---|
| markdown link, autolink, refdef | LOW | MEDIUM |
| markdown image, raw HTML, `data:` URI | LOW | HIGH |

Disposition then applies trust: under `PRESET_USER_UPLOAD` a MEDIUM reaches
QUARANTINE_REVIEW and a HIGH fails secure; under `PRESET_TRUSTED_SOURCE` both warn.
**MEDIUM never hard-fails under either preset** — a query-carrying *link* is held or
warned, never blocked. Two consumers inferred otherwise; it is pinned in
`tests/test_wiring.py`.

Known over-blocks measured against real corpora are listed in
[`LIMITATIONS.md`](LIMITATIONS.md).