docs(url-shape): make the rule reconstructable, and record what three corpora measured
Three consumers reconstructed is_ordinary_url from prose we sent in coordination messages and each produced a different wrong number on a real corpus: one omitted the base64 20-char floor and fired on path words like /blog/; one omitted the opaque-token condition entirely and undercounted; one computed Shannon entropy over whole filenames instead of tokens and concluded the 4.4 floor over-blocks ordinary documents. Same cause each time -- our prose described the rules without their tokenizer. docs/URL-SHAPE.md states the algorithm in order, spells out the separator class and all three length floors, and lists the three reconstruction errors as worked counter-examples. Its example table is parsed and asserted against the real predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the code -- all 18 rows verified load-bearing. LIMITATIONS.md brought current with the field measurements: - Percent-escape is no longer zero. Two English corpora measured 0; a 389-file Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring corpora being English was a sampling bias invisible from inside. - The query over-block now has THREE disjoint benign populations: utm_* tracking, content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version selector. No parameter-level remedy covers any two, which moves this from a conclusion to a settled constraint on 0.4.0. - Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise precise (no other FP in 2401 distinct URLs) and stays. - Raw HTML with a relative URL attribute is HIGH though it reaches no external host, and end tags are counted. - OKF frontmatter: a one-key block-sequence item is silently misparsed to a string where two keys hard-reject, so a pointer can ride past the resource allowlist. Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since both backward-breaking migration targets are nested. Fail-secure, but a compatibility wall that needs a deliberate parse-safety decision. - A persist gate cannot cover execution risk, and that boundary is unowned. New behaviour claims are pinned by tests so a closed concession fails and forces this doc to be updated. 593 -> 631 passed.
This commit is contained in:
parent
956c835d38
commit
684ce3a45f
6 changed files with 383 additions and 18 deletions
66
tests/test_url_shape_doc.py
Normal file
66
tests/test_url_shape_doc.py
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
"""The URL-shape doc is executable: every worked example is asserted against code.
|
||||
|
||||
`docs/URL-SHAPE.md` exists because three consumers reconstructed `is_ordinary_url`
|
||||
from prose and each got a different wrong answer. A reference that can drift from
|
||||
the implementation would reproduce exactly that failure, so the worked-examples
|
||||
table is parsed out of the document and run through the real predicate here.
|
||||
|
||||
A row that disagrees with the code fails this test — whichever of the two is wrong.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from llm_ingestion_guard.active_content import is_ordinary_url
|
||||
from llm_ingestion_guard.entropy import is_base64_like, is_hex_blob
|
||||
from llm_ingestion_guard import calibration as cal
|
||||
|
||||
_DOC = Path(__file__).resolve().parent.parent / "docs" / "URL-SHAPE.md"
|
||||
|
||||
# `| `<url>` | <verdict> | <why> |` — the verdict column is the assertion.
|
||||
_ROW_RE = re.compile(r"^\|\s*`([^`]+)`\s*\|\s*(ordinary|carrying)\s*\|", re.MULTILINE)
|
||||
|
||||
|
||||
def _rows() -> list[tuple[str, bool]]:
|
||||
text = _DOC.read_text(encoding="utf-8")
|
||||
return [(url, verdict == "ordinary") for url, verdict in _ROW_RE.findall(text)]
|
||||
|
||||
|
||||
def test_doc_exists_and_table_was_actually_parsed():
|
||||
# Without this floor a renamed heading or reformatted table would empty the
|
||||
# parametrize list and turn the whole file into a silent pass.
|
||||
rows = _rows()
|
||||
assert len(rows) >= 15, f"only parsed {len(rows)} worked examples from {_DOC.name}"
|
||||
assert any(ordinary for _, ordinary in rows), "no ordinary examples parsed"
|
||||
assert any(not ordinary for _, ordinary in rows), "no carrying examples parsed"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("url,expected_ordinary", _rows(), ids=[u for u, _ in _rows()])
|
||||
def test_worked_example_matches_implementation(url, expected_ordinary):
|
||||
assert is_ordinary_url(url) is expected_ordinary
|
||||
|
||||
|
||||
def test_documented_floors_match_calibration():
|
||||
# The floors table in the doc states three numbers. They are the reason every
|
||||
# reconstruction that omitted them over-fired, so they are pinned to source.
|
||||
text = _DOC.read_text(encoding="utf-8")
|
||||
assert "≥ 20 chars" in text and "≥ 32 chars" in text and "≥ 24 chars" in text
|
||||
assert f"≥ **{cal.URL_OPAQUE_ENTROPY_H}**" in text
|
||||
assert cal.URL_OPAQUE_MIN_LEN == 24 and cal.URL_OPAQUE_HEX_MIN_LEN == 32
|
||||
# The base64 floor lives in `entropy`, not `calibration` — assert behaviourally.
|
||||
assert is_base64_like("A" * 20) and not is_base64_like("A" * 19)
|
||||
assert is_hex_blob("a" * 32) and not is_hex_blob("a" * 31)
|
||||
|
||||
|
||||
def test_documented_separator_class_matches_the_tokenizer():
|
||||
# The doc spells out the separator characters because omitting tokenization was
|
||||
# the error that produced the largest wrong number. Keep the two in step.
|
||||
from llm_ingestion_guard.active_content import _URL_TOKEN_RE
|
||||
|
||||
text = _DOC.read_text(encoding="utf-8")
|
||||
assert _URL_TOKEN_RE.pattern in text, "tokenizer regex not quoted verbatim in the doc"
|
||||
for sep in "/._-~+,;:=&$!*'()":
|
||||
assert _URL_TOKEN_RE.split(f"a{sep}b") == ["a", "b"], sep
|
||||
Loading…
Add table
Add a link
Reference in a new issue