open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity

docs(url-shape): make the rule reconstructable, and record what three corpora measured

Browse source 
Three consumers reconstructed is_ordinary_url from prose we sent in coordination
messages and each produced a different wrong number on a real corpus: one omitted
the base64 20-char floor and fired on path words like /blog/; one omitted the
opaque-token condition entirely and undercounted; one computed Shannon entropy
over whole filenames instead of tokens and concluded the 4.4 floor over-blocks
ordinary documents. Same cause each time -- our prose described the rules without
their tokenizer.

docs/URL-SHAPE.md states the algorithm in order, spells out the separator class
and all three length floors, and lists the three reconstruction errors as worked
counter-examples. Its example table is parsed and asserted against the real
predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the
code -- all 18 rows verified load-bearing.

LIMITATIONS.md brought current with the field measurements:

- Percent-escape is no longer zero. Two English corpora measured 0; a 389-file
  Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just
  o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring
  corpora being English was a sampling bias invisible from inside.
- The query over-block now has THREE disjoint benign populations: utm_* tracking,
  content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version
  selector. No parameter-level remedy covers any two, which moves this from a
  conclusion to a settled constraint on 0.4.0.
- Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise
  precise (no other FP in 2401 distinct URLs) and stays.
- Raw HTML with a relative URL attribute is HIGH though it reaches no external
  host, and end tags are counted.
- OKF frontmatter: a one-key block-sequence item is silently misparsed to a string
  where two keys hard-reject, so a pointer can ride past the resource allowlist.
  Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since
  both backward-breaking migration targets are nested. Fail-secure, but a
  compatibility wall that needs a deliberate parse-safety decision.
- A persist gate cannot cover execution risk, and that boundary is unowned.

New behaviour claims are pinned by tests so a closed concession fails and forces
this doc to be updated. 593 -> 631 passed.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-27 08:56:24 +02:00
commit 684ce3a45f
6 changed files with 383 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

104
docs/LIMITATIONS.md
View file

@ -40,6 +40,34 @@ items; this is the full list, each with the mechanism.
(an injection in a directory listing is caught) rather than path-rejecting the
conformant bundle. A front-end materialising individual uploads keeps the opposite
rule (`allow_reserved=False`): a reserved basename is a listing-shadow and refused.
- **OKF frontmatter is a restricted grammar, and a one-key block-sequence item is
silently misparsed.** Gate T2 accepts a line-oriented subset deliberately — full
YAML is a larger parse-attack surface than a write-time gate needs. Nested mappings
and flow collections (`[a, b]`, `{k: v}`) are *rejected outright*, which fails
secure. The defect is between those two outcomes: a block sequence whose items carry
exactly **one** key parses "successfully" into the wrong type —
`sources:\n - uri: https://e.com/a` yields the **string** `'uri: https://e.com/a'`,
not a mapping, while the same list with two keys per item hard-rejects. A pointer
can therefore ride through in a key the `resource` allowlist never inspects
(`attester:\n - resource: attesters/sql_equality.py` → WARN), whereas a top-level
`resource:` with a relative path correctly fails secure. The shape is not conformant
OKF, so a well-formed bundle will not produce it; a malformed or hostile one can, and
mode-b `import_bundle` writes the merged concept verbatim.
- **Consequence: an OKF v0.2 concept cannot traverse the external-import path.** Both
of v0.2's backward-breaking migration targets are nested — `timestamp` → `generated.at`,
and body `# Citations` → a `sources` block list of mappings — so a conformant v0.2
concept fails secure at the frontmatter gate. This is the correct direction but it is
a compatibility wall, not a policy: v0.2 support requires a deliberate parse-safety
decision about widening the grammar, and the dangling-or-substituted `executor`/
`attester` pointer question only becomes live once that decision is made.
- **A persist gate cannot cover execution risk.** OKF v0.2 introduces concepts whose
purpose is to *name code to be run* (`runtime`, `executor.resource`,
`attester.resource`). This library answers "is this safe to **store**"; executable
code carries its risk at **run**. A file that is harmless to persist can be harmful
to point at. Upstream defers the attester ABI and sandboxing to a future revision, so
there is no runtime contract to gate against — the execution boundary is *unowned*
across the stack rather than covered by anyone's roadmap, and no tightening of a
write-time scanner would change that.
- **A document that *describes* attacks is a false positive.** Content documenting
prompt-injection payloads (security notes, this project's own corpus) trips
carrier-strip / fail-secure. At the text layer "*about* an attack" and "*carrying*
@ -92,27 +120,71 @@ items; this is the full list, each with the mechanism.
QUARANTINE_REVIEW / FAIL_SECURE on an untrusted upload. Obfuscated encoding is a
core exfil primitive and the ambiguous case is put on the review side deliberately;
it is listed here because it is the same *class* of over-block that 0.3.1 fixed,
in a rarer shape. **Measured (2026-07-25, two independent consumer corpora):
zero occurrences** — 0 of 347 external URLs in a 527-document vendor-docs corpus,
0 of 81 in a real capture store. That *bounds* the shape as rare in linked content;
it does not retire it. A consumer deriving filenames from titles through a slugger
produces `%20` systematically rather than incidentally, and that corpus is still
unmeasured — so the open question is narrower than it was, not closed.
in a rarer shape. **It is a non-ASCII-language tax, and that is the finding.**
Three consumer corpora measured it (2026-07-25/26). The two English ones found
zero — 0 of 347 external URLs in a 527-document vendor-docs corpus, 0 of 81 in a
capture store. The third, a 389-file Norwegian/Microsoft reference corpus, found
10 distinct real escapes and **every one of them Norwegian**: `%C3%B8` and
`%C3%A5` are simply *ø* and *å* in UTF-8, and legal/government sources turn titles
into paths (`lovdata.no/…/kap2/%C2%A710`). "Accepted false positive" reads
differently as "URLs in your own language grade above LOW". Our two zero-measuring
corpora were both English; that was a sampling bias invisible from inside.
Two sub-classes worth separating: `{tenant}`/`{agent-id}` template placeholders in
API code samples encode to `%7B`/`%7D` (26 of that corpus's 36 hits — an artifact of
harvesting, not of prose), and a slugger deriving filenames from titles would
produce escapes *systematically* rather than incidentally — still unmeasured.
- **A percent-escaped path also defeats tokenization, which feeds the entropy
branch.** `%` is not in the separator class, so `NSMs%20Grunnprinsipper%20for%20IKT`
is one 34-character token where the same title with literal spaces would be four
short ones. Measured at H=4.04 — under the 4.4 floor, and moot in practice because
the `%` rule already disqualifies the URL. It is recorded because it means the
length floor does less work in non-ASCII paths than the calibration assumed.
- **Legitimate CDN content-asset ids trip the hex branch, permanently.** A ≥32-char
hex path segment is opaque by design, and several public CMSes mint asset URLs that
way (measured: 9 distinct on `regjeringen.no`, `ks.no`, `datatilsynet.no`). This is
the "legitimate build hash or doc id" case the calibration predicted, now confirmed
present in the field. The class does not decay — it is how those systems generate
URLs — so it is a standing false positive rather than a transient one. The branch is
otherwise precise (no other false positives in 2401 distinct URLs) and stays.
- **A non-empty query is graded as data-carrying — the over-block that actually
occurs in the field.** The same two corpora found this rule firing on *disjoint*
benign populations: 16 of 16 query-carrying external URLs in the vendor-docs corpus
were publisher-authored campaign tracking (`utm_*` on the publisher's own domains),
while 35 of 35 in the capture store were content identity (`?v=`, `?channel_id=`,
`?all=true`) where the parameter *is* the resource. **No parameter-level remedy
covers both:** an allowlist keyed on tracking-parameter names resolves the first
population entirely and the second not at all, and stripping the query is lossless
for the first while dereferencing nothing for the second. The cost is bounded — a
occurs in the field.** Three corpora have now measured it, and each found a
*disjoint* benign population:
**(1)** 16 of 16 query-carrying external URLs in a vendor-docs corpus were
publisher-authored campaign tracking (`utm_*` on the publisher's own domains);
**(2)** 28 of 28 in a capture store were content identity (`?v=`, `?channel_id=`,
`?all=true`) where the parameter *is* the resource; **(3)** 149 of 1694 distinct
`learn.microsoft.com` URLs in a reference corpus carried `?view=`, Microsoft Learn's
own documentation-version selector, plus 23 `?api-version=`. **No parameter-level
remedy covers any two of them, let alone all three:** an allowlist keyed on
tracking-parameter names resolves (1) entirely and (2) and (3) not at all; stripping
the query is lossless for (1), dereferences nothing for (2), and silently changes
*which document is cited* for (3) — the worst failure mode of the three, because the
result stays plausible. This is a settled constraint on any future middle tier, not
a hypothesis. The cost is bounded — a
query-carrying *link* is MEDIUM, so it disposes QUARANTINE_REVIEW under
`PRESET_USER_UPLOAD` and WARN under `PRESET_TRUSTED_SOURCE`: held or warned, never
hard-failed (pinned in `tests/test_wiring.py`, because both consumers inferred a
hard block rather than running it). An *image* keeps HIGH, and that is the carrier
where this would bite — neither corpus contained a single remote image, so the
image row of this limitation remains unmeasured in the field.
where this would bite — the two corpora that reported a carrier breakdown contained
zero remote images and the third did not report one, so the image row of this
limitation remains unmeasured in the field.
- **Raw HTML with a *relative* URL attribute is HIGH, though it can reach no
attacker-controlled host.** The markdown paths test for an external target before
flagging; the raw-HTML path deliberately does not, because an active element needs
no URL at all (an `on*=` handler executes on its own). That reasoning covers event
handlers but over-reaches on the URL-attribute branch: an element outside the active
name set carrying `href="/en/agent-sdk/quickstart"` — an internal doc route — grades
HIGH. Measured on a vendor-docs corpus, where it lands on MDX components:
`<Card href="/…">` fires this way, and `<Frame>` fires on the *name* branch alone
because names are lower-cased and `frame` is in the active set — legacy HTML
framesets, which appear in essentially no modern documentation, while `Frame` is a
common component name. Case is not an available discriminator: HTML is
case-insensitive, so PascalCase cannot be treated as "component, not tag".
- **Raw-HTML findings count end tags.** `</a>` is active by name on its own, so a
corpus census that counts only opening tags understates what this detector reports
by roughly the ratio of closing to opening active tags (measured at 1.6× on one
corpus). Severity and finding count are unaffected — the class collapses to one
finding — but the `count` field is not a document count.
- **URL fragments are not graded.** A fragment is never sent to the server, so it
cannot carry data to the host a renderer auto-fetches, and `…/overview#section` is
the most common shape in real documentation. The residual: a *clicked* link to an

121
docs/URL-SHAPE.md Normal file
View file

@ -0,0 +1,121 @@
# The URL-shape rule, stated so it can be reconstructed
`is_ordinary_url` decides whether a URL merely *names* a remote document or can
*carry data outward*. Severity grades on that answer (0.3.1), so a consumer
reasoning about its own corpus needs the rule exactly, not approximately.
**This document exists because approximate statements of it failed three times in
two days.** Three independent consumers reconstructed the rule from prose we sent
in coordination messages, and each produced a wrong number in a different way —
see [Commonly reconstructed wrong](#commonly-reconstructed-wrong). The worked
examples below are asserted against the real implementation by
`tests/test_url_shape_doc.py`, so this file cannot drift from the code.
## The algorithm, in order
A URL is **ordinary** only if every step passes. Any failure means *carrying*.
1. **Scheme gate.** `^(?:https?://|//)` — case-insensitive. Only http(s) and
protocol-relative URLs have an ordinary form at all. Every other scheme
(`javascript:`, `data:`, `ftp:`, `file:`, …) is active or fetches out-of-band on
its own terms and never grades down.
2. **Parse.** `urlsplit(url)`. A `ValueError` (malformed authority — bad IPv6, bad
port) is *not ordinary*: unparseable means ungradable, and ungradable fails closed.
3. **Query and userinfo.** `parts.query`, `parts.username`, `parts.password` — any
non-empty value means carrying. A query can move a value to the host; userinfo
is a credential in the URL.
4. **Percent-escapes.** `"%" in (parts.netloc + parts.path)` means carrying.
5. **Opaque tokens.** Split `parts.netloc + parts.path` on the separator class and
test **every** token. Any opaque token means carrying.
The **fragment is deliberately excluded** from all of this. It is never sent to the
server, so it cannot carry data to a host that a renderer auto-fetches, and
`…/overview#prerequisites` is the single most common shape in real documentation.
### Step 5 in full — the part that is always missed
Tokenization happens **first**, and the entropy/base64/hex tests apply to each
resulting token, **never to the whole path or filename**:
```python
_URL_TOKEN_RE = re.compile(r"[/._\-~+,;:=&$!*'()]+")
tokens = [t for t in _URL_TOKEN_RE.split(parts.netloc + parts.path) if t]
```
Note what is in that character class: **`/` `.` `_` `-` `~` `+` `,` `;` `:` `=` `&`
`$` `!` `*` `'` `(` `)`**. Hyphens, underscores and dots are separators, so a long
hyphenated filename shatters into short tokens. Note also what is *not* in it:
`%` is not a separator, so a percent-escaped path yields *longer* tokens than the
same path with literal spaces.
A token is **opaque** if any of three branches fires — cheapest first:
| Branch | Floor | Test |
|---|---|---|
| base64 | **≥ 20 chars** | matches `[A-Za-z0-9+/]{20,}={0,3}` **and** b64-decodes to ≥80% printable text |
| hex | **≥ 32 chars** | matches `(?:0x)?[0-9a-fA-F]{32,}` |
| entropy | **≥ 24 chars** | Shannon entropy ≥ **4.4** |
Every branch has a length floor. Nothing shorter than 20 characters can ever be
opaque by any branch.
## Worked examples
Asserted against `is_ordinary_url` by `tests/test_url_shape_doc.py`. `ordinary`
means "names a document" (LOW); `carrying` keeps the carrier's full severity.
| URL | Verdict | Why |
|---|---|---|
| `https://learn.microsoft.com/en-us/azure/overview` | ordinary | bare path |
| `https://learn.microsoft.com/en-us/azure/overview#prerequisites` | ordinary | fragment excluded |
| `https://youtu.be/dQw4w9WgXcQ` | ordinary | 11 chars, under every floor |
| `https://www.bbc.com/news/articles/c8x9k2m1l0po` | ordinary | 12 chars, under every floor |
| `https://example.com/a/550e8400-e29b-41d4-a716-446655440000/doc` | ordinary | UUID splits on `-`; longest token 12 |
| `https://example.com/docs/CISPE-Buying-Cloud-Services-in-Public-Sector-Handbook-v2-FEB-2022_EN-Source_v2_Norwegian.pdf` | ordinary | 16 tokens, longest 9, worst H=3.17 |
| `https://ai.meta.com/blog/` | ordinary | `blog` is 4 chars — below the base64 floor |
| `https://www.youtube.com/watch?v=dQw4w9WgXcQ` | carrying | non-empty query |
| `https://claude.com/pricing?utm_source=docs` | carrying | non-empty query |
| `https://learn.microsoft.com/azure/ai/overview?view=azureml-api-2` | carrying | non-empty query |
| `https://nsm.no/NSMs%20Grunnprinsipper%20for%20IKT-sikkerhet.pdf` | carrying | percent-escapes |
| `https://lovdata.no/nav/lov/2025-06-20-81/kap2/%C2%A710` | carrying | percent-escapes |
| `https://token:s3cr3t@evil.test/p.png` | carrying | userinfo |
| `https://example.com/d41d8cd98f00b204e9800998ecf8427e` | carrying | 32-char hex token |
| `https://evil.test/c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU/p.png` | carrying | base64 decoding to text |
| `https://c3RvbGVuIHNlc3Npb24gdG9rZW4gdmFsdWU.evil.test/p.png` | carrying | opaque host label |
| `ftp://example.com/pub/file.txt` | carrying | scheme gate |
| `javascript:alert(1)` | carrying | scheme gate |
## Commonly reconstructed wrong
Each of these produced a real, wrong number on a real corpus.
- **Omitting tokenization.** Computing Shannon entropy over the whole filename
scores `CISPE-Buying-…-Norwegian.pdf` at H=4.75 and concludes the 4.4 floor
over-blocks ordinary documents. Under the real tokenizer its worst token is
`Norwegian` at H=3.17 and the URL is ordinary. Entropy is **per token**.
- **Omitting the base64 length floor.** Testing "b64-decodes to printable text"
without `len ≥ 20` fires on ordinary path words — `blog`, `digi` and `1544`
decode to printable bytes, while `news` and `docs` do not, so the reconstruction
appears to discriminate on something meaningful and does not.
- **Omitting step 5 entirely.** Applying only "query or percent-escape" is
*strictly weaker* than the real rule, so it undercounts carrying URLs. It can
never overcount: every URL it calls carrying really is.
## Grading, once the shape is known
Shape decides *severity*, not disposition. Per-construct severities live in
`calibration.py`; an ordinary URL grades `ACTIVE_CONTENT_ORDINARY_SEVERITY` (LOW).
| Carrier | Ordinary URL | Carrying URL |
|---|---|---|
| markdown link, autolink, refdef | LOW | MEDIUM |
| markdown image, raw HTML, `data:` URI | LOW | HIGH |
Disposition then applies trust: under `PRESET_USER_UPLOAD` a MEDIUM reaches
QUARANTINE_REVIEW and a HIGH fails secure; under `PRESET_TRUSTED_SOURCE` both warn.
**MEDIUM never hard-fails under either preset** — a query-carrying *link* is held or
warned, never blocked. Two consumers inferred otherwise; it is pinned in
`tests/test_wiring.py`.
Known over-blocks measured against real corpora are listed in
[`LIMITATIONS.md`](LIMITATIONS.md).