open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity

docs(url-shape): make the rule reconstructable, and record what three corpora measured

Browse source 
Three consumers reconstructed is_ordinary_url from prose we sent in coordination
messages and each produced a different wrong number on a real corpus: one omitted
the base64 20-char floor and fired on path words like /blog/; one omitted the
opaque-token condition entirely and undercounted; one computed Shannon entropy
over whole filenames instead of tokens and concluded the 4.4 floor over-blocks
ordinary documents. Same cause each time -- our prose described the rules without
their tokenizer.

docs/URL-SHAPE.md states the algorithm in order, spells out the separator class
and all three length floors, and lists the three reconstruction errors as worked
counter-examples. Its example table is parsed and asserted against the real
predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the
code -- all 18 rows verified load-bearing.

LIMITATIONS.md brought current with the field measurements:

- Percent-escape is no longer zero. Two English corpora measured 0; a 389-file
  Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just
  o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring
  corpora being English was a sampling bias invisible from inside.
- The query over-block now has THREE disjoint benign populations: utm_* tracking,
  content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version
  selector. No parameter-level remedy covers any two, which moves this from a
  conclusion to a settled constraint on 0.4.0.
- Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise
  precise (no other FP in 2401 distinct URLs) and stays.
- Raw HTML with a relative URL attribute is HIGH though it reaches no external
  host, and end tags are counted.
- OKF frontmatter: a one-key block-sequence item is silently misparsed to a string
  where two keys hard-reject, so a pointer can ride past the resource allowlist.
  Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since
  both backward-breaking migration targets are nested. Fail-secure, but a
  compatibility wall that needs a deliberate parse-safety decision.
- A persist gate cannot cover execution risk, and that boundary is unowned.

New behaviour claims are pinned by tests so a closed concession fails and forces
this doc to be updated. 593 -> 631 passed.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-27 08:56:24 +02:00
commit 684ce3a45f
6 changed files with 383 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

121
docs/URL-SHAPE.md Normal file
View file

@ -0,0 +1,121 @@
# The URL-shape rule, stated so it can be reconstructed
`is_ordinary_url` decides whether a URL merely *names* a remote document or can
*carry data outward*. Severity grades on that answer (0.3.1), so a consumer
reasoning about its own corpus needs the rule exactly, not approximately.
**This document exists because approximate statements of it failed three times in
two days.** Three independent consumers reconstructed the rule from prose we sent
in coordination messages, and each produced a wrong number in a different way —
see [Commonly reconstructed wrong](#commonly-reconstructed-wrong). The worked
examples below are asserted against the real implementation by
`tests/test_url_shape_doc.py`, so this file cannot drift from the code.
## The algorithm, in order
A URL is **ordinary** only if every step passes. Any failure means *carrying*.
1. **Scheme gate.** `^(?:https?://|//)` — case-insensitive. Only http(s) and
protocol-relative URLs have an ordinary form at all. Every other scheme
(`javascript:`, `data:`, `ftp:`, `file:`, …) is active or fetches out-of-band on
its own terms and never grades down.
2. **Parse.** `urlsplit(url)`. A `ValueError` (malformed authority — bad IPv6, bad
port) is *not ordinary*: unparseable means ungradable, and ungradable fails closed.
3. **Query and userinfo.** `parts.query`, `parts.username`, `parts.password` — any
non-empty value means carrying. A query can move a value to the host; userinfo
is a credential in the URL.
4. **Percent-escapes.** `"%" in (parts.netloc + parts.path)` means carrying.
5. **Opaque tokens.** Split `parts.netloc + parts.path` on the separator class and
test **every** token. Any opaque token means carrying.
The **fragment is deliberately excluded** from all of this. It is never sent to the
server, so it cannot carry data to a host that a renderer auto-fetches, and
`…/overview#prerequisites` is the single most common shape in real documentation.
### Step 5 in full — the part that is always missed
Tokenization happens **first**, and the entropy/base64/hex tests apply to each
resulting token, **never to the whole path or filename**:
```python
_URL_TOKEN_RE = re.compile(r"[/._\-~+,;:=&$!*'()]+")
tokens = [t for t in _URL_TOKEN_RE.split(parts.netloc + parts.path) if t]
```
Note what is in that character class: **`/` `.` `_` `-` `~` `+` `,` `;` `:` `=` `&`
`$` `!` `*` `'` `(` `)`**. Hyphens, underscores and dots are separators, so a long
hyphenated filename shatters into short tokens. Note also what is *not* in it:
`%` is not a separator, so a percent-escaped path yields *longer* tokens than the
same path with literal spaces.
A token is **opaque** if any of three branches fires — cheapest first:
| Branch | Floor | Test |
|---|---|---|
| base64 | **≥ 20 chars** | matches `[A-Za-z0-9+/]{20,}={0,3}` **and** b64-decodes to ≥80% printable text |
| hex | **≥ 32 chars** | matches `(?:0x)?[0-9a-fA-F]{32,}` |
| entropy | **≥ 24 chars** | Shannon entropy ≥ **4.4** |
Every branch has a length floor. Nothing shorter than 20 characters can ever be
opaque by any branch.
## Worked examples
Asserted against `is_ordinary_url` by `tests/test_url_shape_doc.py`. `ordinary`
means "names a document" (LOW); `carrying` keeps the carrier's full severity.
| URL | Verdict | Why |
|---|---|---|
| `https://learn.microsoft.com/en-us/azure/overview` | ordinary | bare path |
| `https://learn.microsoft.com/en-us/azure/overview#prerequisites` | ordinary | fragment excluded |
| `https://youtu.be/dQw4w9WgXcQ` | ordinary | 11 chars, under every floor |
| `https://www.bbc.com/news/articles/c8x9k2m1l0po` | ordinary | 12 chars, under every floor |
| `https://example.com/a/550e8400-e29b-41d4-a716-446655440000/doc` | ordinary | UUID splits on `-`; longest token 12 |
| `https://example.com/docs/CISPE-Buying-Cloud-Services-in-Public-Sector-Handbook-v2-FEB-2022_EN-Source_v2_Norwegian.pdf` | ordinary | 16 tokens, longest 9, worst H=3.17 |
| `https://ai.meta.com/blog/` | ordinary | `blog` is 4 chars — below the base64 floor |
| `https://www.youtube.com/watch?v=dQw4w9WgXcQ` | carrying | non-empty query |
| `https://claude.com/pricing?utm_source=docs` | carrying | non-empty query |
| `https://learn.microsoft.com/azure/ai/overview?view=azureml-api-2` | carrying | non-empty query |
| `https://nsm.no/NSMs%20Grunnprinsipper%20for%20IKT-sikkerhet.pdf` | carrying | percent-escapes |
| `https://lovdata.no/nav/lov/2025-06-20-81/kap2/%C2%A710` | carrying | percent-escapes |
| `https://token:s3cr3t@evil.test/p.png` | carrying | userinfo |
| `https://example.com/d41d8cd98f00b204e9800998ecf8427e` | carrying | 32-char hex token |
| `https://evil.test/c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU/p.png` | carrying | base64 decoding to text |
| `https://c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU.evil.test/p.png` | carrying | opaque host label |
| `ftp://example.com/pub/file.txt` | carrying | scheme gate |
| `javascript:alert(1)` | carrying | scheme gate |
## Commonly reconstructed wrong
Each of these produced a real, wrong number on a real corpus.
- **Omitting tokenization.** Computing Shannon entropy over the whole filename
scores `CISPE-Buying-…-Norwegian.pdf` at H=4.75 and concludes the 4.4 floor
over-blocks ordinary documents. Under the real tokenizer its worst token is
`Norwegian` at H=3.17 and the URL is ordinary. Entropy is **per token**.
- **Omitting the base64 length floor.** Testing "b64-decodes to printable text"
without `len ≥ 20` fires on ordinary path words — `blog`, `digi` and `1544`
decode to printable bytes, while `news` and `docs` do not, so the reconstruction
appears to discriminate on something meaningful and does not.
- **Omitting step 5 entirely.** Applying only "query or percent-escape" is
*strictly weaker* than the real rule, so it undercounts carrying URLs. It can
never overcount: every URL it calls carrying really is.
## Grading, once the shape is known
Shape decides *severity*, not disposition. Per-construct severities live in
`calibration.py`; an ordinary URL grades `ACTIVE_CONTENT_ORDINARY_SEVERITY` (LOW).
| Carrier | Ordinary URL | Carrying URL |
|---|---|---|
| markdown link, autolink, refdef | LOW | MEDIUM |
| markdown image, raw HTML, `data:` URI | LOW | HIGH |
Disposition then applies trust: under `PRESET_USER_UPLOAD` a MEDIUM reaches
QUARANTINE_REVIEW and a HIGH fails secure; under `PRESET_TRUSTED_SOURCE` both warn.
**MEDIUM never hard-fails under either preset** — a query-carrying *link* is held or
warned, never blocked. Two consumers inferred otherwise; it is pinned in
`tests/test_wiring.py`.
Known over-blocks measured against real corpora are listed in
[`LIMITATIONS.md`](LIMITATIONS.md).