A consumer corrected a claim we shipped this morning. We wrote that both
zero-measuring corpora were English, and used that to explain the third corpus's
10 Norwegian hits as a sampling bias. One of those two is a Norwegian repo with
Norwegian content, so the stated explanation was wrong.
Their measurement separates two axes we had conflated. For third-party LINK
corpora -- URLs collected from other people's sites -- language does predict, and
the Norwegian legal/government sources are where the escapes were. For GENERATED
paths the predictor is slugger class: a whitelist slugger ([^a-z0-9]+ -> "-")
cannot emit an escape in any language because it discards the character before
anything encodes it, while encodeURIComponent produces them systematically as soon
as titles are non-ASCII. Their own slugger returns a structural zero on Norwegian
input, not a statistical one.
So non-ASCII language is a confounder for encode-vs-whitelist, and slugger class is
the thing that is actually testable at a consumer -- a one-line code read rather
than a corpus census. Reframed accordingly, with the earlier conflation named so
the correction is visible rather than silently rewritten.
Kjell Tore Guttormsen
910a12455a