docs(url-shape): make the rule reconstructable, and record what three corpora measured
Three consumers reconstructed is_ordinary_url from prose we sent in coordination messages and each produced a different wrong number on a real corpus: one omitted the base64 20-char floor and fired on path words like /blog/; one omitted the opaque-token condition entirely and undercounted; one computed Shannon entropy over whole filenames instead of tokens and concluded the 4.4 floor over-blocks ordinary documents. Same cause each time -- our prose described the rules without their tokenizer. docs/URL-SHAPE.md states the algorithm in order, spells out the separator class and all three length floors, and lists the three reconstruction errors as worked counter-examples. Its example table is parsed and asserted against the real predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the code -- all 18 rows verified load-bearing. LIMITATIONS.md brought current with the field measurements: - Percent-escape is no longer zero. Two English corpora measured 0; a 389-file Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring corpora being English was a sampling bias invisible from inside. - The query over-block now has THREE disjoint benign populations: utm_* tracking, content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version selector. No parameter-level remedy covers any two, which moves this from a conclusion to a settled constraint on 0.4.0. - Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise precise (no other FP in 2401 distinct URLs) and stays. - Raw HTML with a relative URL attribute is HIGH though it reaches no external host, and end tags are counted. - OKF frontmatter: a one-key block-sequence item is silently misparsed to a string where two keys hard-reject, so a pointer can ride past the resource allowlist. Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since both backward-breaking migration targets are nested. Fail-secure, but a compatibility wall that needs a deliberate parse-safety decision. - A persist gate cannot cover execution risk, and that boundary is unowned. New behaviour claims are pinned by tests so a closed concession fails and forces this doc to be updated. 593 -> 631 passed.
This commit is contained in:
parent
956c835d38
commit
684ce3a45f
6 changed files with 383 additions and 18 deletions
|
|
@ -259,3 +259,38 @@ def test_default_source_is_output_and_override_respected():
|
|||
for f in scan_active_content(_ECHOLEAK).findings)
|
||||
assert all(f.source is Source.INPUT
|
||||
for f in scan_active_content(_ECHOLEAK, source=Source.INPUT).findings)
|
||||
|
||||
|
||||
# --- raw-HTML over-blocks measured on a vendor-docs corpus (2026-07-26) -------
|
||||
# Documented in docs/LIMITATIONS.md. Pinned so the concessions stay honest: a
|
||||
# closed over-block should fail here and force the doc to be updated.
|
||||
|
||||
@pytest.mark.parametrize("cid,text", [
|
||||
# Fires on the URL-attr branch although the target is a relative doc route,
|
||||
# which cannot reach an attacker-controlled host. `Card` is not in the active
|
||||
# name set — the href alone carries it.
|
||||
("relative-href-on-inactive-name",
|
||||
'<Card title="Quickstart" icon="play" href="/en/agent-sdk/quickstart">'),
|
||||
# Fires on the *name* branch: names are lower-cased and `frame` is in the
|
||||
# active set (legacy HTML framesets), while `Frame` is a common MDX component.
|
||||
("mdx-component-named-like-a-tag", "<Frame>"),
|
||||
])
|
||||
def test_raw_html_overblocks_are_still_high(cid, text):
|
||||
finding = [f for f in scan_active_content(text).findings
|
||||
if f.label == "active:raw-html"]
|
||||
assert len(finding) == 1, f"{cid}: raw-html not reported"
|
||||
assert finding[0].severity is Severity.HIGH, f"{cid}: {finding[0].severity}"
|
||||
|
||||
|
||||
def test_raw_html_counts_end_tags():
|
||||
# `</a>` is active by name on its own, so a corpus census counting only opening
|
||||
# tags understates this detector's `count`. The class still collapses to ONE
|
||||
# finding — the count is what moves.
|
||||
solo = [f for f in scan_active_content("</a>").findings
|
||||
if f.label == "active:raw-html"]
|
||||
assert len(solo) == 1 and solo[0].count == 1
|
||||
|
||||
pair = [f for f in scan_active_content('<a href="https://x.example/p">t</a>').findings
|
||||
if f.label == "active:raw-html"]
|
||||
assert len(pair) == 1, "a start/end pair must not split into two findings"
|
||||
assert pair[0].count == 2, f"end tag not counted: {pair[0].count}"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue