1
0
Fork 0

docs(url-shape): make the rule reconstructable, and record what three corpora measured

Three consumers reconstructed is_ordinary_url from prose we sent in coordination
messages and each produced a different wrong number on a real corpus: one omitted
the base64 20-char floor and fired on path words like /blog/; one omitted the
opaque-token condition entirely and undercounted; one computed Shannon entropy
over whole filenames instead of tokens and concluded the 4.4 floor over-blocks
ordinary documents. Same cause each time -- our prose described the rules without
their tokenizer.

docs/URL-SHAPE.md states the algorithm in order, spells out the separator class
and all three length floors, and lists the three reconstruction errors as worked
counter-examples. Its example table is parsed and asserted against the real
predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the
code -- all 18 rows verified load-bearing.

LIMITATIONS.md brought current with the field measurements:

- Percent-escape is no longer zero. Two English corpora measured 0; a 389-file
  Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just
  o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring
  corpora being English was a sampling bias invisible from inside.
- The query over-block now has THREE disjoint benign populations: utm_* tracking,
  content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version
  selector. No parameter-level remedy covers any two, which moves this from a
  conclusion to a settled constraint on 0.4.0.
- Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise
  precise (no other FP in 2401 distinct URLs) and stays.
- Raw HTML with a relative URL attribute is HIGH though it reaches no external
  host, and end tags are counted.
- OKF frontmatter: a one-key block-sequence item is silently misparsed to a string
  where two keys hard-reject, so a pointer can ride past the resource allowlist.
  Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since
  both backward-breaking migration targets are nested. Fail-secure, but a
  compatibility wall that needs a deliberate parse-safety decision.
- A persist gate cannot cover execution risk, and that boundary is unowned.

New behaviour claims are pinned by tests so a closed concession fails and forces
this doc to be updated. 593 -> 631 passed.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-27 08:56:24 +02:00
commit 684ce3a45f
6 changed files with 383 additions and 18 deletions

View file

@ -215,8 +215,9 @@ a green scan means safe content. The highest-impact items:
egress, semantic poisoning, trusted-prose lone-HIGH, lexicon dedup (`count=1`),
pure beaconing, and short opaque URL segments.
**Full list — 18 items, each with the mechanism, plus the out-of-scope boundary:**
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md).
**Full list — 26 items, each with the mechanism, plus the out-of-scope boundary:**
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
consumer corpora, including the false positives the URL-shape rule actually produces.
## Out-of-scope (documented boundary)
@ -226,6 +227,10 @@ steganography; query-time / runtime guardrails; semantic factuality verification
## Design & threat model
- [Design brief](docs/BRIEF.md) — what this repo contains and why.
- [URL-shape rule](docs/URL-SHAPE.md) — `is_ordinary_url` stated precisely enough to
reconstruct, with worked examples asserted against the implementation. **Read this
before reasoning about the rule from prose:** three consumers reconstructed it from
summaries and each produced a different wrong number on a real corpus.
- [Build plan](docs/PLAN.md) — module build order and the reuse map.
- [Adoption brief](docs/ADOPTION-BRIEF.md) — wiring the guard into an OKF
second-brain / LLM wiki, and a checklist for *when* to include it.