open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Pull requests Releases Activity

docs(limitations): correct the percent-escape framing -- slugger class, not language

Browse source 
A consumer corrected a claim we shipped this morning. We wrote that both
zero-measuring corpora were English, and used that to explain the third corpus's
10 Norwegian hits as a sampling bias. One of those two is a Norwegian repo with
Norwegian content, so the stated explanation was wrong.

Their measurement separates two axes we had conflated. For third-party LINK
corpora -- URLs collected from other people's sites -- language does predict, and
the Norwegian legal/government sources are where the escapes were. For GENERATED
paths the predictor is slugger class: a whitelist slugger ([^a-z0-9]+ -> "-")
cannot emit an escape in any language because it discards the character before
anything encodes it, while encodeURIComponent produces them systematically as soon
as titles are non-ASCII. Their own slugger returns a structural zero on Norwegian
input, not a statistical one.

So non-ASCII language is a confounder for encode-vs-whitelist, and slugger class is
the thing that is actually testable at a consumer -- a one-line code read rather
than a corpus census. Reframed accordingly, with the earlier conflation named so
the correction is visible rather than silently rewritten.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-27 20:20:45 +02:00
commit 910a12455a
1 changed files with 17 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

22
docs/LIMITATIONS.md
View file

@ -139,12 +139,24 @@ items; this is the full list, each with the mechanism.
10 distinct real escapes and **every one of them Norwegian**: `%C3%B8` and
`%C3%A5` are simply *ø* and *å* in UTF-8, and legal/government sources turn titles
into paths (`lovdata.no/…/kap2/%C2%A710`). "Accepted false positive" reads
differently as "URLs in your own language grade above LOW". Our two zero-measuring
corpora were both English; that was a sampling bias invisible from inside.
Two sub-classes worth separating: `{tenant}`/`{agent-id}` template placeholders in
differently as "URLs in your own language grade above LOW".
**Two distinct axes, which an earlier revision of this file conflated.** For
*third-party link* corpora — URLs an ingester collects from other people's sites —
language does predict: the corpus of Norwegian legal/government sources carried
escapes where two link corpora of mostly English-language domains carried none.
For *generated paths* the predictor is not language but **slugger class**: a
whitelist slugger (`[^a-z0-9]+ → "-"`) cannot emit an escape in any language,
because it discards the character before anything encodes it — measured
structurally, not statistically, by a consumer whose content is Norwegian and whose
slugger output is `"Løkkene i produksjonslinja" → "l-kkene-i-produksjonslinja"`. A
path built with `encodeURIComponent` produces escapes systematically the moment
titles are non-ASCII. So non-ASCII language is a *confounder* for encode-vs-whitelist,
and the slugger class is the testable thing at a consumer — a one-line code read,
not a corpus census. That question is still open for the one consumer whose slugger
we have not seen.
One sub-class worth separating: `{tenant}`/`{agent-id}` template placeholders in
API code samples encode to `%7B`/`%7D` (26 of that corpus's 36 hits — an artifact of
harvesting, not of prose), and a slugger deriving filenames from titles would
produce escapes *systematically* rather than incidentally — still unmeasured.
harvesting, not of prose).
- **A percent-escaped path also defeats tokenization, which feeds the entropy
branch.** `%` is not in the separator class, so `NSMs%20Grunnprinsipper%20for%20IKT`
is one 34-character token where the same title with literal spaces would be four