Kjell Tore Guttormsen
684ce3a45f
Three consumers reconstructed is_ordinary_url from prose we sent in coordination messages and each produced a different wrong number on a real corpus: one omitted the base64 20-char floor and fired on path words like /blog/; one omitted the opaque-token condition entirely and undercounted; one computed Shannon entropy over whole filenames instead of tokens and concluded the 4.4 floor over-blocks ordinary documents. Same cause each time -- our prose described the rules without their tokenizer. docs/URL-SHAPE.md states the algorithm in order, spells out the separator class and all three length floors, and lists the three reconstruction errors as worked counter-examples. Its example table is parsed and asserted against the real predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the code -- all 18 rows verified load-bearing. LIMITATIONS.md brought current with the field measurements: - Percent-escape is no longer zero. Two English corpora measured 0; a 389-file Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring corpora being English was a sampling bias invisible from inside. - The query over-block now has THREE disjoint benign populations: utm_* tracking, content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version selector. No parameter-level remedy covers any two, which moves this from a conclusion to a settled constraint on 0.4.0. - Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise precise (no other FP in 2401 distinct URLs) and stays. - Raw HTML with a relative URL attribute is HIGH though it reaches no external host, and end tags are counted. - OKF frontmatter: a one-key block-sequence item is silently misparsed to a string where two keys hard-reject, so a pointer can ride past the resource allowlist. Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since both backward-breaking migration targets are nested. Fail-secure, but a compatibility wall that needs a deliberate parse-safety decision. - A persist gate cannot cover execution risk, and that boundary is unowned. New behaviour claims are pinned by tests so a closed concession fails and forces this doc to be updated. 593 -> 631 passed.
121 lines
6.6 KiB
Markdown
121 lines
6.6 KiB
Markdown
# The URL-shape rule, stated so it can be reconstructed
|
|
|
|
`is_ordinary_url` decides whether a URL merely *names* a remote document or can
|
|
*carry data outward*. Severity grades on that answer (0.3.1), so a consumer
|
|
reasoning about its own corpus needs the rule exactly, not approximately.
|
|
|
|
**This document exists because approximate statements of it failed three times in
|
|
two days.** Three independent consumers reconstructed the rule from prose we sent
|
|
in coordination messages, and each produced a wrong number in a different way —
|
|
see [Commonly reconstructed wrong](#commonly-reconstructed-wrong). The worked
|
|
examples below are asserted against the real implementation by
|
|
`tests/test_url_shape_doc.py`, so this file cannot drift from the code.
|
|
|
|
## The algorithm, in order
|
|
|
|
A URL is **ordinary** only if every step passes. Any failure means *carrying*.
|
|
|
|
1. **Scheme gate.** `^(?:https?://|//)` — case-insensitive. Only http(s) and
|
|
protocol-relative URLs have an ordinary form at all. Every other scheme
|
|
(`javascript:`, `data:`, `ftp:`, `file:`, …) is active or fetches out-of-band on
|
|
its own terms and never grades down.
|
|
2. **Parse.** `urlsplit(url)`. A `ValueError` (malformed authority — bad IPv6, bad
|
|
port) is *not ordinary*: unparseable means ungradable, and ungradable fails closed.
|
|
3. **Query and userinfo.** `parts.query`, `parts.username`, `parts.password` — any
|
|
non-empty value means carrying. A query can move a value to the host; userinfo
|
|
is a credential in the URL.
|
|
4. **Percent-escapes.** `"%" in (parts.netloc + parts.path)` means carrying.
|
|
5. **Opaque tokens.** Split `parts.netloc + parts.path` on the separator class and
|
|
test **every** token. Any opaque token means carrying.
|
|
|
|
The **fragment is deliberately excluded** from all of this. It is never sent to the
|
|
server, so it cannot carry data to a host that a renderer auto-fetches, and
|
|
`…/overview#prerequisites` is the single most common shape in real documentation.
|
|
|
|
### Step 5 in full — the part that is always missed
|
|
|
|
Tokenization happens **first**, and the entropy/base64/hex tests apply to each
|
|
resulting token, **never to the whole path or filename**:
|
|
|
|
```python
|
|
_URL_TOKEN_RE = re.compile(r"[/._\-~+,;:=&$!*'()]+")
|
|
tokens = [t for t in _URL_TOKEN_RE.split(parts.netloc + parts.path) if t]
|
|
```
|
|
|
|
Note what is in that character class: **`/` `.` `_` `-` `~` `+` `,` `;` `:` `=` `&`
|
|
`$` `!` `*` `'` `(` `)`**. Hyphens, underscores and dots are separators, so a long
|
|
hyphenated filename shatters into short tokens. Note also what is *not* in it:
|
|
`%` is not a separator, so a percent-escaped path yields *longer* tokens than the
|
|
same path with literal spaces.
|
|
|
|
A token is **opaque** if any of three branches fires — cheapest first:
|
|
|
|
| Branch | Floor | Test |
|
|
|---|---|---|
|
|
| base64 | **≥ 20 chars** | matches `[A-Za-z0-9+/]{20,}={0,3}` **and** b64-decodes to ≥80% printable text |
|
|
| hex | **≥ 32 chars** | matches `(?:0x)?[0-9a-fA-F]{32,}` |
|
|
| entropy | **≥ 24 chars** | Shannon entropy ≥ **4.4** |
|
|
|
|
Every branch has a length floor. Nothing shorter than 20 characters can ever be
|
|
opaque by any branch.
|
|
|
|
## Worked examples
|
|
|
|
Asserted against `is_ordinary_url` by `tests/test_url_shape_doc.py`. `ordinary`
|
|
means "names a document" (LOW); `carrying` keeps the carrier's full severity.
|
|
|
|
| URL | Verdict | Why |
|
|
|---|---|---|
|
|
| `https://learn.microsoft.com/en-us/azure/overview` | ordinary | bare path |
|
|
| `https://learn.microsoft.com/en-us/azure/overview#prerequisites` | ordinary | fragment excluded |
|
|
| `https://youtu.be/dQw4w9WgXcQ` | ordinary | 11 chars, under every floor |
|
|
| `https://www.bbc.com/news/articles/c8x9k2m1l0po` | ordinary | 12 chars, under every floor |
|
|
| `https://example.com/a/550e8400-e29b-41d4-a716-446655440000/doc` | ordinary | UUID splits on `-`; longest token 12 |
|
|
| `https://example.com/docs/CISPE-Buying-Cloud-Services-in-Public-Sector-Handbook-v2-FEB-2022_EN-Source_v2_Norwegian.pdf` | ordinary | 16 tokens, longest 9, worst H=3.17 |
|
|
| `https://ai.meta.com/blog/` | ordinary | `blog` is 4 chars — below the base64 floor |
|
|
| `https://www.youtube.com/watch?v=dQw4w9WgXcQ` | carrying | non-empty query |
|
|
| `https://claude.com/pricing?utm_source=docs` | carrying | non-empty query |
|
|
| `https://learn.microsoft.com/azure/ai/overview?view=azureml-api-2` | carrying | non-empty query |
|
|
| `https://nsm.no/NSMs%20Grunnprinsipper%20for%20IKT-sikkerhet.pdf` | carrying | percent-escapes |
|
|
| `https://lovdata.no/nav/lov/2025-06-20-81/kap2/%C2%A710` | carrying | percent-escapes |
|
|
| `https://token:s3cr3t@evil.test/p.png` | carrying | userinfo |
|
|
| `https://example.com/d41d8cd98f00b204e9800998ecf8427e` | carrying | 32-char hex token |
|
|
| `https://evil.test/c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU/p.png` | carrying | base64 decoding to text |
|
|
| `https://c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU.evil.test/p.png` | carrying | opaque host label |
|
|
| `ftp://example.com/pub/file.txt` | carrying | scheme gate |
|
|
| `javascript:alert(1)` | carrying | scheme gate |
|
|
|
|
## Commonly reconstructed wrong
|
|
|
|
Each of these produced a real, wrong number on a real corpus.
|
|
|
|
- **Omitting tokenization.** Computing Shannon entropy over the whole filename
|
|
scores `CISPE-Buying-…-Norwegian.pdf` at H=4.75 and concludes the 4.4 floor
|
|
over-blocks ordinary documents. Under the real tokenizer its worst token is
|
|
`Norwegian` at H=3.17 and the URL is ordinary. Entropy is **per token**.
|
|
- **Omitting the base64 length floor.** Testing "b64-decodes to printable text"
|
|
without `len ≥ 20` fires on ordinary path words — `blog`, `digi` and `1544`
|
|
decode to printable bytes, while `news` and `docs` do not, so the reconstruction
|
|
appears to discriminate on something meaningful and does not.
|
|
- **Omitting step 5 entirely.** Applying only "query or percent-escape" is
|
|
*strictly weaker* than the real rule, so it undercounts carrying URLs. It can
|
|
never overcount: every URL it calls carrying really is.
|
|
|
|
## Grading, once the shape is known
|
|
|
|
Shape decides *severity*, not disposition. Per-construct severities live in
|
|
`calibration.py`; an ordinary URL grades `ACTIVE_CONTENT_ORDINARY_SEVERITY` (LOW).
|
|
|
|
| Carrier | Ordinary URL | Carrying URL |
|
|
|---|---|---|
|
|
| markdown link, autolink, refdef | LOW | MEDIUM |
|
|
| markdown image, raw HTML, `data:` URI | LOW | HIGH |
|
|
|
|
Disposition then applies trust: under `PRESET_USER_UPLOAD` a MEDIUM reaches
|
|
QUARANTINE_REVIEW and a HIGH fails secure; under `PRESET_TRUSTED_SOURCE` both warn.
|
|
**MEDIUM never hard-fails under either preset** — a query-carrying *link* is held or
|
|
warned, never blocked. Two consumers inferred otherwise; it is pinned in
|
|
`tests/test_wiring.py`.
|
|
|
|
Known over-blocks measured against real corpora are listed in
|
|
[`LIMITATIONS.md`](LIMITATIONS.md).
|