Kjell Tore Guttormsen
684ce3a45f
Three consumers reconstructed is_ordinary_url from prose we sent in coordination messages and each produced a different wrong number on a real corpus: one omitted the base64 20-char floor and fired on path words like /blog/; one omitted the opaque-token condition entirely and undercounted; one computed Shannon entropy over whole filenames instead of tokens and concluded the 4.4 floor over-blocks ordinary documents. Same cause each time -- our prose described the rules without their tokenizer. docs/URL-SHAPE.md states the algorithm in order, spells out the separator class and all three length floors, and lists the three reconstruction errors as worked counter-examples. Its example table is parsed and asserted against the real predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the code -- all 18 rows verified load-bearing. LIMITATIONS.md brought current with the field measurements: - Percent-escape is no longer zero. Two English corpora measured 0; a 389-file Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring corpora being English was a sampling bias invisible from inside. - The query over-block now has THREE disjoint benign populations: utm_* tracking, content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version selector. No parameter-level remedy covers any two, which moves this from a conclusion to a settled constraint on 0.4.0. - Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise precise (no other FP in 2401 distinct URLs) and stays. - Raw HTML with a relative URL attribute is HIGH though it reaches no external host, and end tags are counted. - OKF frontmatter: a one-key block-sequence item is silently misparsed to a string where two keys hard-reject, so a pointer can ride past the resource allowlist. Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since both backward-breaking migration targets are nested. Fail-secure, but a compatibility wall that needs a deliberate parse-safety decision. - A persist gate cannot cover execution risk, and that boundary is unowned. New behaviour claims are pinned by tests so a closed concession fails and forces this doc to be updated. 593 -> 631 passed.
66 lines
3 KiB
Python
66 lines
3 KiB
Python
"""The URL-shape doc is executable: every worked example is asserted against code.
|
|
|
|
`docs/URL-SHAPE.md` exists because three consumers reconstructed `is_ordinary_url`
|
|
from prose and each got a different wrong answer. A reference that can drift from
|
|
the implementation would reproduce exactly that failure, so the worked-examples
|
|
table is parsed out of the document and run through the real predicate here.
|
|
|
|
A row that disagrees with the code fails this test — whichever of the two is wrong.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from llm_ingestion_guard.active_content import is_ordinary_url
|
|
from llm_ingestion_guard.entropy import is_base64_like, is_hex_blob
|
|
from llm_ingestion_guard import calibration as cal
|
|
|
|
_DOC = Path(__file__).resolve().parent.parent / "docs" / "URL-SHAPE.md"
|
|
|
|
# `| `<url>` | <verdict> | <why> |` — the verdict column is the assertion.
|
|
_ROW_RE = re.compile(r"^\|\s*`([^`]+)`\s*\|\s*(ordinary|carrying)\s*\|", re.MULTILINE)
|
|
|
|
|
|
def _rows() -> list[tuple[str, bool]]:
|
|
text = _DOC.read_text(encoding="utf-8")
|
|
return [(url, verdict == "ordinary") for url, verdict in _ROW_RE.findall(text)]
|
|
|
|
|
|
def test_doc_exists_and_table_was_actually_parsed():
|
|
# Without this floor a renamed heading or reformatted table would empty the
|
|
# parametrize list and turn the whole file into a silent pass.
|
|
rows = _rows()
|
|
assert len(rows) >= 15, f"only parsed {len(rows)} worked examples from {_DOC.name}"
|
|
assert any(ordinary for _, ordinary in rows), "no ordinary examples parsed"
|
|
assert any(not ordinary for _, ordinary in rows), "no carrying examples parsed"
|
|
|
|
|
|
@pytest.mark.parametrize("url,expected_ordinary", _rows(), ids=[u for u, _ in _rows()])
|
|
def test_worked_example_matches_implementation(url, expected_ordinary):
|
|
assert is_ordinary_url(url) is expected_ordinary
|
|
|
|
|
|
def test_documented_floors_match_calibration():
|
|
# The floors table in the doc states three numbers. They are the reason every
|
|
# reconstruction that omitted them over-fired, so they are pinned to source.
|
|
text = _DOC.read_text(encoding="utf-8")
|
|
assert "≥ 20 chars" in text and "≥ 32 chars" in text and "≥ 24 chars" in text
|
|
assert f"≥ **{cal.URL_OPAQUE_ENTROPY_H}**" in text
|
|
assert cal.URL_OPAQUE_MIN_LEN == 24 and cal.URL_OPAQUE_HEX_MIN_LEN == 32
|
|
# The base64 floor lives in `entropy`, not `calibration` — assert behaviourally.
|
|
assert is_base64_like("A" * 20) and not is_base64_like("A" * 19)
|
|
assert is_hex_blob("a" * 32) and not is_hex_blob("a" * 31)
|
|
|
|
|
|
def test_documented_separator_class_matches_the_tokenizer():
|
|
# The doc spells out the separator characters because omitting tokenization was
|
|
# the error that produced the largest wrong number. Keep the two in step.
|
|
from llm_ingestion_guard.active_content import _URL_TOKEN_RE
|
|
|
|
text = _DOC.read_text(encoding="utf-8")
|
|
assert _URL_TOKEN_RE.pattern in text, "tokenizer regex not quoted verbatim in the doc"
|
|
for sep in "/._-~+,;:=&$!*'()":
|
|
assert _URL_TOKEN_RE.split(f"a{sep}b") == ["a", "b"], sep
|