docs(url-shape): make the rule reconstructable, and record what three corpora measured
Three consumers reconstructed is_ordinary_url from prose we sent in coordination messages and each produced a different wrong number on a real corpus: one omitted the base64 20-char floor and fired on path words like /blog/; one omitted the opaque-token condition entirely and undercounted; one computed Shannon entropy over whole filenames instead of tokens and concluded the 4.4 floor over-blocks ordinary documents. Same cause each time -- our prose described the rules without their tokenizer. docs/URL-SHAPE.md states the algorithm in order, spells out the separator class and all three length floors, and lists the three reconstruction errors as worked counter-examples. Its example table is parsed and asserted against the real predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the code -- all 18 rows verified load-bearing. LIMITATIONS.md brought current with the field measurements: - Percent-escape is no longer zero. Two English corpora measured 0; a 389-file Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring corpora being English was a sampling bias invisible from inside. - The query over-block now has THREE disjoint benign populations: utm_* tracking, content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version selector. No parameter-level remedy covers any two, which moves this from a conclusion to a settled constraint on 0.4.0. - Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise precise (no other FP in 2401 distinct URLs) and stays. - Raw HTML with a relative URL attribute is HIGH though it reaches no external host, and end tags are counted. - OKF frontmatter: a one-key block-sequence item is silently misparsed to a string where two keys hard-reject, so a pointer can ride past the resource allowlist. Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since both backward-breaking migration targets are nested. Fail-secure, but a compatibility wall that needs a deliberate parse-safety decision. - A persist gate cannot cover execution risk, and that boundary is unowned. New behaviour claims are pinned by tests so a closed concession fails and forces this doc to be updated. 593 -> 631 passed.
This commit is contained in:
parent
956c835d38
commit
684ce3a45f
6 changed files with 383 additions and 18 deletions
|
|
@ -215,8 +215,9 @@ a green scan means safe content. The highest-impact items:
|
|||
egress, semantic poisoning, trusted-prose lone-HIGH, lexicon dedup (`count=1`),
|
||||
pure beaconing, and short opaque URL segments.
|
||||
|
||||
**Full list — 18 items, each with the mechanism, plus the out-of-scope boundary:**
|
||||
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md).
|
||||
**Full list — 26 items, each with the mechanism, plus the out-of-scope boundary:**
|
||||
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
|
||||
consumer corpora, including the false positives the URL-shape rule actually produces.
|
||||
|
||||
## Out-of-scope (documented boundary)
|
||||
|
||||
|
|
@ -226,6 +227,10 @@ steganography; query-time / runtime guardrails; semantic factuality verification
|
|||
## Design & threat model
|
||||
|
||||
- [Design brief](docs/BRIEF.md) — what this repo contains and why.
|
||||
- [URL-shape rule](docs/URL-SHAPE.md) — `is_ordinary_url` stated precisely enough to
|
||||
reconstruct, with worked examples asserted against the implementation. **Read this
|
||||
before reasoning about the rule from prose:** three consumers reconstructed it from
|
||||
summaries and each produced a different wrong number on a real corpus.
|
||||
- [Build plan](docs/PLAN.md) — module build order and the reuse map.
|
||||
- [Adoption brief](docs/ADOPTION-BRIEF.md) — wiring the guard into an OKF
|
||||
second-brain / LLM wiki, and a checklist for *when* to include it.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue